How do I start reading from a specific line in a text file

[cpt_jack]

Hi helpful people

I have been having trouble with a large text file (8mb) of almost 1 million lines. I am currently using stream_get_line() to insert each line to my DB, the file i'm working with gets updated every hour but the existing data doesn't change it just gets new lines added on the end so i thought if i could count how many lines were in my DB then start stream_get_line() from the next line in the file i could skip most of the work and just insert the new lines. Problem is i can't find anything that i can use to start that function at the beginning of a specific line. here is my code so far;

p.s. i put a bit in that makes it only insert the new lines and not the old ones but it still does the stream_get_line() for every line and that is the bit that takes the time.

 

function read_line($file, $line_num, $delimiter="\n")

{

$fp = fopen($file, 'r');

$count = 0;

while(!feof($fp)) {

$buffer = stream_get_line($fp, 100, $delimiter);

list($village_id, $timestamp, $new_owner, $old_owner) = explode(',', $buffer);

if($count >= $line_num) {

mysql_query("INSERT INTO conquers SET village_id='$village_id', timestamp='$timestamp', new_owner='$new_owner', old_owner='$old_owner', valid= 1 ON DUPLICATE KEY UPDATE valid= 1");

}

$count++;

}

}

$file = "http://en5.tribalwars.net/map/conquer.txt";

dbconnect(5);

$result1 = mysql_query("SELECT valid FROM conquers");

$line_num = mysql_num_rows($result1);

dbclose();

dbconnect(5);

read_line($file, $line_num);

dbclose();

[Lithium]

Re: How do I start reading from a specific line in a text file

you need to have the file with the complete full insert's or you need it only hourly (the text one)? if you only need it hourly, why not deleting the file after dumping the data on to the db?

[cpt_jack]

Re: How do I start reading from a specific line in a text file

the idea is each hour the file has X amount of new lines at the end of it and i add the lines to my DB. so it starts with a file of 10 lines, i add those, next hour it has 20 lines the first 10 being the original that i now have stored so i am now only interested in the new 10 ones. does that make sense?

[cpt_jack]

Re: How do I start reading from a specific line in a text file

No ideas anybody?

I tried using fseek() to move the pointer through the file then use stream_get_line() but either i did it wrong or it doesn't work like i thought it did.

also fseek() moves the pointer to a specific byte not a line and since each line has a different amount of bytes i don't know how to tell it where to start.

[Lithium]

Re: How do I start reading from a specific line in a text file

 

the idea is each hour the file has X amount of new lines at the end of it and i add the lines to my DB. so it starts with a file of 10 lines, i add those, next hour it has 20 lines the first 10 being the original that i now have stored so i am now only interested in the new 10 ones. does that make sense?

It does make sense, and yet my opinion is still that you should either delete or empty the file after passing the data to the db, also, for the file itself it seems you have timestamps there so why not fetching data with the specific timestamp and the lines that follow?

[cpt_jack]

Re: How do I start reading from a specific line in a text file

 

It does make sense, and yet my opinion is still that you should either delete or empty the file after passing the data to the db, also, for the file itself it seems you have timestamps there so why not fetching data with the specific timestamp and the lines that follow?

Oh sorry i misunderstood you, the file i get each hour comes from another server and i have no control over what is in it. which means i can't remove the old data....see my problem

EDIT i think i just got what you meant, i don't know why i didn't think of it before. i copy the file to my server, delete all the old lines then just insert what is rest.......i'm such an idiot for not thinking that :-) i will try that approach and let you know what happens.