[Xmldatadumps-l] Important news about the November dumps run!

[Ariel Glenn WMF]

As was previously announced on the xmldatadumps-l list, the sql/xml dumps
generated twice a month will be written to an internal server, starting
with the November run. This is in part to reduce load on the web/rsync/nfs
server which has been doing this work also until now. We want separation
of roles for some other reasons too.

Because I want to get this right, and there are a lot of moving parts, and
I don't want to rsync all the prefetch data over to these boxes again next
month after cancelling the move:

********
If needed, the November full run will be delayed for a few days.
If the November full run takes too long, the partial run, usually starting
on the 20th of the month, will not take place.
*********

Additionally, as described in an earlier email on the xmldatadumps-l list:

*********
files will show up on the web server/rsync server with a substantial
delay. Initially this may be a day or more. This includes index.html and
other status files.
*********

You can keep track of developments here:
https://phabricator.wikimedia.org/T178893

If you know folks not on the lists in the recipients field for this email,
please forward it to them and suggest that they subscribe to this list.

Thanks,

Ariel

[Ariel Glenn WMF]

The first set of dumps is running there and looks like it's working ok.
I've done a manual rsync of files produced up to this point, so those are
now available on the web server.

As before, you can follow work on this at
https://phabricator.wikimedia.org/T178893

Note that it is possible that some index.html files may contain links to
files which did not get picked up on the rsync. They'll be there sometime
tomorrow after the next rsync.

Ariel

Post by Ariel Glenn WMF
As was previously announced on the xmldatadumps-l list, the sql/xml dumps
generated twice a month will be written to an internal server, starting
with the November run. This is in part to reduce load on the web/rsync/nfs
server which has been doing this work also until now. We want separation
of roles for some other reasons too.
Because I want to get this right, and there are a lot of moving parts, and
I don't want to rsync all the prefetch data over to these boxes again next
********
If needed, the November full run will be delayed for a few days.
If the November full run takes too long, the partial run, usually starting
on the 20th of the month, will not take place.
*********
*********
files will show up on the web server/rsync server with a substantial
delay. Initially this may be a day or more. This includes index.html and
other status files.
*********
You can keep track of developments here: https://phabricator.wikimedia.
org/T178893
If you know folks not on the lists in the recipients field for this email,
please forward it to them and suggest that they subscribe to this list.
Thanks,
Ariel

[Ariel Glenn WMF]

Rsync of xml/sql dumps to the web server is now running on a rolling basis
via a script, so you should see updates regularly rather than "every
$random hours". There's more to be done on that front, see
https://phabricator.wikimedia.org/T179857 for what's next.

Ariel