Reply to comment

How would you handle the upload of very big files on a distant ftp server ?
Or , put differently, how would you notice on a distant ftp folder that big files arrived in full , and are complete , if you are not the one that send the file.
The code below might not work on a window- based machine, i only use it on linux servers, and didn't try it on a personnal computer , so don't ask me to resolve problems between windows and Apache/PHP or that kind of stuff, you won't find me ....

Anyway , this kind of problematic is really related to programmers in a professional environment, so i don't think i'll have a lot of people reading it, still, if it can be of any help !!
The situation is : you have an access to a FTP server, it could be a public server, for instance ( see http://mambo.ucsc.edu/psl/sgiftp.html for a list of servers ) , from which you have to retrieve data from time to time , or a client send you files for instance videos on a FTP . How would you know that the data you' re retrieving is complete , how would you adapt your batch process so that it starts downloading the stuff that is apparently complete ?
There's a way to do that, using FTP commands, and the code i propose is a batch in PHP .

// folder on the FTP that is scrutinized
$pathftp = "/home/www/html/wires/efe/import";

//  folder on my linux machine where the files will be downloaded to
define('TEMPFOLDER','usr/monrepertoire/');


// connection to the FTP
$hostftpdistant = "FTPHOST" ;
$ftp_user_name = 'USER' ;
$ftp_user_pass = 'LOGIN';
$conn_distant = ftp_connect($hostftpdistant);
if(!$login_distant = @ftp_login($conn_distant, $ftp_user_name, $ftp_user_pass)){
        logerror('connexion ftp  impossible') ;
        print  "Operation failed Login /connexion pb to ".$hostftpdistant." , please try again  ".$rc;
        die();
}
ftp_chdir($conn_distant,$pathftp) ;

// tables : res for the current table, prev for the previous one (the one generated by the batch last time it ran),
// to_process is the table i use to store the names of the files that are apparently complete.
// The two first tables have an identical structure to compare them .
$res = array() ;
$prev  = array();
$to_process = array();


//  List all the files in the distant FTP folder.
$buff = ftp_rawlist($conn_id, '.');
 

..This make up a table of lines like this .
Array
(
[0] => -rw-r--r-- 1 507 507 7787422 Oct 06 15:04 Envio_1_20091006_1600.zip
[1] => -rw-r--r-- 1 507 507 11020588 Oct 06 15:34 Envio_1_20091006_1630.zip
[2] => -rw-r--r-- 1 507 507 11921288 Oct 06 16:05 Envio_1_20091006_1700.zip

)

my fonction parse_rawlist_into_array transform this first table onto a table of key values for each element

$res = parse_rawlist_into_array($buff ,$path);

the final result is a table in which each element has for key the name of the file .I keep the complete path to the file, it size ( that will be used to compare and see if the file is complete ), and its date ..

[Envio_1_20091007_0000.zip] => Array
(
[path] => /home/www/html/wires/efe/import
[size] => 7812960
[month] => Oct
[day] => 06
[name] => Envio_1_20091007_0000.zip
)

Now you can read the .txt that lists all the files from the precedent run of the cron and that were not treated.

$prev = parse_textfile_into_array(FOLDER.'fichiers_present.txt',$path);
//  compare the size of the file . If equals, i put them in the table  to_process .
// If these are different, i suppose the upload is not finished yet, so i write the name of the file in the .txt fichiers_present.txt .
foreach($res as $filetocheck){
        if($filetocheck['size'] ==  $prev[$filetocheck['name']]['size'] ){
                $to_process[] = $filetocheck['name'] ;
        }
}


foreach($to_process as $file){
                debugConsole('Retrieve a zip  ZIP : '.$file) ;
                if(!$fp = fopen(TEMPFOLDER.$file,'w') ){
                        die('problem rights on the folder '.TEMPFOLDER);
                }

                if(!ftp_fget($conn_distant,$fp, $file ,FTP_BINARY)){
                        logerror('impossible to retrieve the file '.$file) ;
                        $db_ext->disconnect();
                        $db->disconnect();
                        fclose($fp);
                        die();
                }
                fclose($fp);                         
                // delete the file from the remote folder once it's done.
                if (!ftp_delete($conn_distant, $file)) {
                        logerror('impossible to delete the file on distant server '.$file) ;                                
                }
}

// then i rewrite in the  text file the files that were not retrieved
// I do that with  ftp_rawlist
$newbuff = parse_rawlist($buff, $path);
if(!($handle = fopen(FOLDER.'fichiers_present.txt',"w+")))
    die("
Cannot open \"".$file."\".
\n\n");

And that's about it , now you can cfix the frequency of your batch using crontab, for instance every 10 minutes .

Now the three functions that i wrote for this script that return the arrays :

function parse_rawlist_into_array($array,$path)
{
    $structure = array();
    foreach($array as $curraw)
    {
        $struc = array();
        $current = preg_split("/[\s]+/",$curraw,10);
        // on prend seulement les fichiers pas les directories
        if($current[4]>0){
            $struc['path'] = $path;
            $struc['size']  = $current[4];
            $struc['month']  = $current[5];
            $struc['day']    = $current[6];
            $struc['name'] = $current[8];
            $structure[$struc['name']] = $struc;
        }
    }
   return $structure;

}
function parse_textfile_into_array($file,$path){
    if(!($handle = fopen($file,"a")))
            die("
Cannot open \"".$file."\".
\n\n");

    $struc = array();
    $structure = array();
    $lines = file($file);
    foreach($lines as $Ligne){
        $current = preg_split("/[\s]+/",$Ligne,10);
        $struc['path'] = $path;
        $struc['size']  = $current[4];
        $struc['month']  = $current[5];
        $struc['day']    = $current[6];
        $current[9] ?  $struc['name'] = $current[9] : $struc['name'] = $current[8] ;
        $structure[$struc['name']] = $struc;
    }
    fclose($handle);
    return $structure ;
}
function parse_rawlist($array,$path)
{
    foreach($array as $Ligne){
        $current = preg_split("/[\s]+/",$Ligne,9);
        if($current[4]>0){
            $buffer .= $Ligne."\n";
        }
    }
    return $buffer;
}