Merge Win32 branch

[chris]

The changes from the  General branch need to be merged into trunk, or rejected.

Merge via the  Merge branch for review and testing.

[chris]

Please review  http://bbdev.fluffy.co.uk/trac/changeset/817

[chris]

Please review  http://bbdev.fluffy.co.uk/trac/changeset/814

[chris]

Please review  http://bbdev.fluffy.co.uk/trac/changeset/798

[chris]

Please review  http://bbdev.fluffy.co.uk/trac/changeset/749

[chris]

Please review  http://bbdev.fluffy.co.uk/trac/changeset/819

[chris]

Please review [821]

[chris]

Please review [825]

[chris]

Please review [827]

[chris]

Please review [829]

[chris]

Please review [831]

[chris]

Please review 832

[chris]

Please review [833]

[martin]

[817] ok to merge

[814] ok to merge

[798] ok to merge but lines 277 to 283 (catch BoxException?) are redundant because BoxException? is derived from std::exception.

[749] ok to merge

[chris]

Please review [835]

[chris]

Thanks Martin, I've merged 817, 814, 798 (with change) and 749.

[martin]

Ok to merge: [821] [825] [827] [829] [831] [832]

[martin]

Ok to merge: [833] [835]

Excepting [819] queried on -dev, up to date at this point.

[chris]

Thanks again Martin, please review [840].

[chris]

All merged to date except [819] and [840].

[chris]

Please review [860]

[chris]

Please review [861]

[chris]

(In [863]) * bin/bbstored/BackupStoreDaemon.h

• Reinstate SendMessageToHousekeepingProcess?() on Win32, but make it do nothing (refs #3)

[chris]

Please review [862] and [863]

[chris]

(In [864]) * configure.ac

Revert to trunk (refs #3)

[chris]

(In [867]) * docs/backup/win32_build_on_linux_using_mingw.txt (refs #3)

Updated documentation for building Win32 native builds on Linux

[chris]

(In [869]) * infrastructure/BoxPlatform.pm.in

Determine whether we are building for Windows or not, and export that information

Search for VERSION.txt in a few more places, for example when running Perl scripts like getversion.pl

(refs #3)

[chris]

(In [871]) * infrastructure/buildenv-testmain-template.cpp

Record the file and line of first test failure, and print them at the end of the test, useful for debugging when the first failure has scrolled off screen (refs #3)

[chris]

(In [873]) * infrastructure/makebuildenv.pl.in

Support multiple implicit dependencies

Add lib/win32 as an implicit dependency on Win32

Make lib/common depend on lib/win32 on Win32

Fix space between test file name platform executable extension, and between executable name and arguments, in test shell scripts

Add support for compiling resource files (.rc) on Win32

Fix dependencies to avoid unnecessary rebuilds when not using BSD make

Code cleanups

Fix typos

(refs #3)

[chris]

(In [875]) * lib/backupclient/BackupClientFileAttributes.cpp

Don't call geteuid() on Win32, since it's emulated, always returns 0, and I want to remove it entirely (refs #3)

[chris]

(In [877]) * lib/backupclient/BackupStoreFile.cpp

Close file before trying to apply attributes to it. Otherwise, when we close it the timestamp will be updated on Win32 (refs #3)

[chris]

(In [879]) * lib/backupclient/BackupStoreObjectDump.cpp

Fix format strings on Win32 (refs #3)

[chris]

Please review [881]

[chris]

(In [883]) Use DIRECTORY_SEPARATOR instead of assuming that it's a forward slash (refs #3)

[chris]

(In [885]) Disable intercept tests on Win32 as well

Define O_BINARY to 0 (zero) if our platform doesn't have it (all except Win32?) which enables us to reduce #ifdefs

(refs #3)

[chris]

(In [887]) Make FDGETLINE_BUFFER_SIZE big enough for one unicode character on Win32, otherwise reading from console can fail due to insufficient buffer size. (refs #3)

[chris]

(In [889]) Add O_BINARY to default flags, since most files opened this way should be opened in binary mode on Win32 (refs #3)

[chris]

(In [892]) Disable all calls to set*id() on Win32 (doesn't work) (refs #3)

[chris]

(In [894]) (refs #3)

Don't include headers that we don't have

Open RAID files in binary mode

[chris]

(In [896]) (refs #3)

Open files in binary mode (Win32)

Disable the lock failure block when we don't have any locking mechanism

Close and delete files before renaming over them on Win32. This breaks Ben's desired recovery semantics, so it's not done on other platforms, but Win32 requires it.

[martin]

Ok to merge: [860] [863] [881]

[861] is ok except for the unnecessary call to Close as mailed.

[862] is ok but now the file is being deleted twice (once inside the try/catch block and once after). Should remove the one outside the try/catch.

Still outstanding: [840]

[chris]

(In [898]) (refs #3)

Initialise Windows sockets automatically for all daemons on Win32

Write PID files on Win32

[chris]

(In [900]) (refs #3)

64-bit format fixes (Win32)

[chris]

(In [902]) (refs #3)

Added an OnIdle? method which can be overridden by subclasses for idle tasks. Used for housekeeping on Win32.

Avoid forking on Win32, and trying to clean up after children.

[chris]

Sorry, please review [865] instead of [864]

[chris]

(In [904]) (refs #3)

Define a suitable constant for invalid socket handles, for use in SocketStream?.cpp

[chris]

Please review [906]

[chris]

(In [908]) (refs #3)

Added support for non-blocking sockets on Win32 as well

[chris]

(In [910]) (refs #3)

Use strict and warnings for code safety.

Exit with status 2 on invalid usage.

Allow running multiple tests, comma separated.

Exit with status 2 if make fails for any test.

Exit with status 1 if any test fails.

[chris]

(In [912]) (refs #3)

Use truncation/sparse files to quickly create files with 200MB of zeroes

[chris]

(In [913]) (refs #3)

64-bit format string fixes for Win32

[chris]

(In [914]) (refs #3)

Use CopyFile? to copy on Windows, instead of shelling out to cp, which doesn't understand native paths.

[chris]

(In [915]) (refs #3)

Close open file handles on Win32

[chris]

(In [916]) (refs #3)

Completely disable symlink test on Win32

[chris]

(In [917]) (refs #3)

Cosmetic fixes to comments

[chris]

(In [918]) (refs #3)

This test takes a long time on Win32 (slow file access? VMware?), so:

• Disable verbose debug logging, which makes it even slower
• Print a warning before the slow test starts
• Increase timeout

[chris]

(In [920]) (refs #3)

Close before unlink

Check that unlink succeeds

[chris]

(In [921]) (refs #3)

We cannot open multiple connections to the server on Win32, so work around.

[chris]

(In [922]) (refs #3)

Use correct directory separator everywhere

[chris]

(In [923]) (refs #3)

Disable symlink tests on Win32

[chris]

(In [927]) (refs #3)

Convert UNIX paths to native on Win32 (avoids #ifdefs in tests)

[chris]

(In [928]) (refs #3)

Disable tests for memory leaks after killing bbstored on Win32, since it dies forcibly without a chance to write the memory leak file.

Initialise Windows sockets on Win32

Add tests for multiply open files on Win32

[chris]

(In [930]) (refs #3)

Remove newlines from output to avoid failure to recognise it on Win32

[chris]

(In [931]) (refs #3)

Added a RunCommand?() function which converts UNIX paths to native before calling ::system() on Win32.

[chris]

(In [932]) (refs #3)

Call RunCommand?() instead of ::system to handle native paths better

[chris]

(In [934]) (refs #3)

Initialise Windows sockets on Win32

[chris]

(In [935]) (refs #3)

Fix format strings on Win32

[chris]

(In [936]) (refs #3)

Cannot signal bbstored to housekeep immediately on Win32, so just wait for it to happen.

[chris]

(In [937]) (refs #3)

Don't check for memory leaks on Win32, since the process is force killed and doesn't have a chance to write the log file.

[chris]

(In [938]) (refs #3)

Use RunCommand? instead of ::system to convert UNIX paths to native

[chris]

(In [940]) (refs #3)

Sleep without nanosleep() on Win32

[chris]

(In [941]) (refs #3)

Automatically initialise Windows sockets for all tests

[chris]

(In [942]) (refs #3)

Reorganise typedefs for clarity

[chris]

(In [943]) (refs #3)

Remove dependency on Box.h and hence on lib/common

[chris]

(In [944]) (refs #3)

Add message definitions and resource files

[chris]

(In [945]) (refs #3)

Include message resource definitions

[chris]

(In [946]) (refs #3)

Add BSD-licensed getopt.h to replace our own version

[chris]

(In [947]) (refs #3)

Use our own ConvertUtf8ToWideString for simplicity

[chris]

(In [948]) (refs #3)

Convert UNIX to native paths

[chris]

(In [949]) (refs #3)

Return INVALID_HANDLE_VALUE instead of NULL from openfile() on failure

[chris]

(In [950]) (refs #3)

Use INVALID_HANDLE_VALUE instead of NULL for invalid handles on Win32

[chris]

(In [951]) (refs #3)

Improved handling of UNIX flags to openfile(), allows multiple opens

[chris]

(In [952]) (refs #3)

Improved error handling in emu_fstat

[chris]

(In [953]) (refs #3)

Always fill in st_uid, st_gid and st_nlink

Return zero size for directories

Improved emulation of Unix file modes (helps when restoring Windows files on Unix)

[chris]

(In [954]) (refs #3)

Pass requested open flags to OpenFileByNameUtf8

[chris]

(In [955]) (refs #3)

Request READ_CONTROL when standard open fails, rather than no flags

[chris]

(In [956]) (refs #3)

Treat PATH_NOT_FOUND just like FILE_NOT_FOUND in Unix land (errno = ENOENT)

[chris]

(In [957]) (refs #3)

Added emulated utimes, chmod, readv and writev

Added useful utility functions ConvertFileTimeToTime_t and ConvertTime_tToFileTime

[chris]

(In [958]) (refs #3)

Removed last vestiges of old getopt()

[chris]

(In [959]) (refs #3)

Improved poll() emulation to handle multiple fds

[chris]

(In [960]) (refs #3)

Fixed looking up message source in syslog()

Flush stdout after writing to it

Allow syslog() to log to console even when openlog() has not been called

[chris]

(In [961]) (refs #3)

Emulated chdir, mkdir and unlink should handle file names in UTF-8 as well

[chris]

(In [962]) (refs #3)

Emulate readdir's d_type field

[chris]

(In [963]) (refs #3)

Remove more vestiges of old getopt

[chris]

(In [964]) (refs #3)

Added prototypes for new emulated functions emu_utimes, readv and writev

[chris]

(In [965]) (refs #3)

Remove inline ConvertFileTimeToTime_t, add prototypes for new versions in emu.cpp

[chris]

(In [966]) (refs #3)

Changed WideSize? from int to size_t

Don't allow space for null terminator that will not be added

[chris]

(In [968]) (refs #3)

Fixed typo

[chris]

Hi Martin,

Currently outstanding as I understand it:

• [819] waiting for me to implement InvisibleTempFileStream?
• [840] waiting for Ben to be satisfied that there are sufficient tests of this functionality
• [843] waiting for me to rewrite to re-enable locking on Win32
• [860] approved, already applied?
• [861] waiting on me to remove #ifdefs
• [862] needs InvisibleTempFileStream?
• [863] approved, already applied?
• [865]
• [867]
• [869]
• [871]
• [873]
• [875] rejected
• [877]
• [879]
• [881] approved, already applied?
• [883]
• [885]
• [887]
• [889]
• [892] rejected, but the comment spelling fix should be applied
• [894]
• [896] rejected, needs a rewrite as Ben suggested
• [898]
• [900]
• [902]
• [904]
• [906]
• [908]
• [910]
• [912]
• [913]
• [914]
• [915]
• [916]
• [917]
• [918]
• [920]
• [921]
• [922] rejected DIRECTORY_SEPARATOR, but keep the f.Close();
• [923]
• [927]
• [928]
• [930]
• [931]
• [934]
• [935]
• [936]
• [937]
• [938]
• [940]
• [941]
• [942]
• [943]
• [944]
• [945]
• [946]
• [947]
• [948]
• [949]
• [950]
• [951]
• [952]
• [953]
• [954]
• [955]
• [956]
• [957] and [964]
• [958] and [963]
• [959]
• [960]
• [961]
• [962]
• [965]
• [966]
• [968]

[martin]

[865] ok [867] ok [869] please use $OSNAME (probably need 'use English' too), $^{O is horrible, go ahead & merge when you've made that change [871] ok, or you could give -l 10000 to xterm :-) (maybe not so easy in mswin) [873] ok [877] ok [879] ok [883] ok [885] ok [887] ok [889] ok [894] ok}

[martin]

(Just reformatting)

• [865] ok
• [867] ok
• [869] please use $OSNAME (probably need 'use English' too), $^O is horrible, go ahead & merge when you've made that change
• [871] ok, or you could give -l 10000 to xterm :-) (maybe not so easy in mswin)
• [873] ok
• [877] ok
• [879] ok
• [883] ok
• [885] ok
• [887] ok
• [889] ok
• [894] ok

[chris]

(In [981]) * Close file before trying to apply attributes to it. Otherwise, when we close it the timestamp will be updated on Win32 (refs #3)

[chris]

Thanks Martin! Please ignore [981].

Merged [865] [867] [869] [871] [873] [877] [879] [883] [885] [887] [889] [894] into trunk.

[chris]

Merged [861] [863] [881] to trunk.

[martin]

• [898] ok
• [900] ok
• [902] ok
• [904] & [906] ok (these should have been one patch, both implementing the same change)
• [908] ok, though ifdefs such as this one would be better based on the feature (ie. presence of fcntl/O_NONBLOCK) rather than the platform. We'll need to revisit these later.
• [910] ok
• [912] ok
• [913] ok
• [914] ok
• [915] ok
• [916] ok
• [917] ok
• [918] to discuss
• [920] ok
• [921] ok

[chris]

(In [995]) * Added support for non-blocking sockets on Win32 as well (TODO revisit #ifdefs: refs #3)

[chris]

Thanks Martin!

Merged [898] [900] [902] [904] [906] [908] [910] [912] [913] [914] [915] [916] [917] [920] [921] to trunk.

[chris]

Please review [1013] [1014] [1015] [1016] for merge.

[chris]

Please review [1017] for merge.

[chris]

(In [1019]) * Comment grammar fix (refs #3)

[chris]

Sorry, please ignore [1019].

[chris]

(In [1022]) * Use readdir's d_type field to determine type of directory entry without

statting the entry, which allows users to exclude unreadable entries to suppress warnings about them on Win32 (refs #3)

[chris]

(In [1026]) * Apply Box coding standards (refs #3)

[chris]

(In [1027]) * Replace global daemon object with a pointer, to allow deletion and

clean up reported memory leaks

• No need to initialise Winsock here, now that lib/server/Daemon does it for us
• Initialise logging properly (refs #3)

[chris]

(In [1029]) * Apply Box coding standards (refs #3, replaces [1026])

[chris]

(In [1030]) * Catch invalid output from sync allow script and tell the user what it

was, to help them debug the problem (refs #3)

[chris]

(In [1031]) * Replace global daemon object with a pointer, to allow deletion and

clean up reported memory leaks (refs #3, combine with [1027])

[chris]

(In [1032]) * Operator precedence fix (refs #3)

[chris]

(In [1035]) * Fix compile error caused by conflicting MSVC macros (refs #3)

[chris]

(In [1036]) * Add option to bbackupquery list command to show times in UTC or local time

(refs #3)

[chris]

(In [1037]) Use BoxPlatform? to see if we are building on Windows (refs #3)

[chris]

(In [1049]) Allow getting an object with a path, not just from the current directory (refs #3)

[chris]

(In [1050]) Convert command-line arguments from the system locale/character set to the console character set (code page), so they they can be converted from console to UTF-8 (yuck).

Don't try to read from stdin or change its code page when it's not open (invalid file handle) (refs #3)

[chris]

(In [1051]) Convert command-line arguments from the system locale/character set to the console character set. (refs #3, combine with [1050])

[chris]

(In [1052]) Add an option to ignore attribute differences on compare. (refs #3)

[chris]

(In [1053]) Compile fix. (refs #3)

[chris]

(In [1054]) Compile fix (refs #3)

[chris]

(In [1055]) Compile fix (refs #3)

[chris]

(In [1058]) Added tests for case-insensitive matching in exclude lists on Win32. (refs #3)

[chris]

(In [1059]) Copy our template config.h to BoxConfig?.h when compiling with MSVC. (refs #3)

[chris]

(In [1061]) Revert patch [862] (refs #3)

[chris]

(In [1062]) Revert patch [825] (mostly).

Re-enable locking code on Win32 (although it's never used).

Use InvisibleTempFileStream? to ensure that the temporary file is always deleted. (refs #3)

[chris]

(In [1064]) Change test for return code 3 back to 2, until or unless the bbackupquery compare error (unreadable files) patch is merged. (refs #3)

[chris]

(In [1070]) Don't check for nanosleep, random device, or flock/F_SETLK on any mingw32 platform (including mingw32msvc). (refs #3)

[chris]

(In [1071]) Don't check for AC_HEADER_DIRENT on any mingw32 platform. (refs #3)

[chris]

(In [1072]) Different check for have_regex_h (the old one doesn't seem to work on MinGW). (refs #3)

[chris]

(In [1073]) Add test/bbackupd/testfiles/syncallowscript.pl to the list of Perl files auto-generated with substitutions. (refs #3)

[chris]

(In [1075]) Rename config.h.win32 to the new name, which will be auto-included by another change (refs #3)

[chris]

(In [1076]) Improved debugging when creation of sparse files fails. (refs #3)

[chris]

(In [1077]) Updated MSVC project files. (refs #3)

[chris]

(In [1078]) Added a script to determine the build version on Win32 MSVC. (refs #3)

[chris]

(In [1079]) Undefine fstat() so that we get the Win32 POSIX version, instead of our emulated version.

Small code cleanup. (refs #3)

[chris]

(In [1080]) Remove #ifdefs around geteuid(), re-enable use of fake version under Win32 as requested by Ben.

Restore serialised AttrModificationTime? under Win32 by passing to emu_utimes(). (refs #3)

[chris]

(In [1081]) Delete existing entries to allow reuse of a BackupStoreDirectory? object. (refs #3)

[chris]

(In [1082]) Add new return codes from BackupClientRestore? for use on Win32. (refs #3)

[chris]

(In [1083]) Use overlapped I/O to avoid blocking.

Use INVALID_HANDLE_VALUE instead of NULL everywhere to avoid bugs, and for type safety. (refs #3)

[chris]

(In [1084]) Implement LocalProcessStream? on Win32. (refs #3)

[chris]

(In [1085]) Use -1 for INVALID_SOCKET_HANDLE to ensure that it's always less than zero, so that non-conditional code in SocketStream?.cpp works properly.

Add a method to determine if a socket stream has been attached to a socket yet. (refs #3)

[chris]

(In [1086]) Use more accurate sleeps in poll() to ensure that we don't end up busy waiting for the last fraction of a second with repeated poll(..., 0). (refs #3)

[chris]

(In [1087]) Use BoxConfig?-MSVC.h and BoxVersion?.h when compiling with MSVC. (refs #3)

[chris]

(In [1088]) Fix memory corruption when the number of not-leaks exceeds the table size. (refs #3)

[chris]

(In [1089]) Match case-insensitively on Win32. (refs #3)

[chris]

(In [1090]) Set the filename to "HANDLE" under Win32 when initialising from an existing handle.

Handle ERROR_BROKEN_PIPE as EOF when using FileStream? to read from a pipe in LocalProcessStream?. (refs #3)

[chris]

(In [1091]) Add a new CommonException? type, AccessDenied? (very useful for debugging access errors, and needed by Boxi to report a sensible error to the user). (refs #3)

[chris]

(In [1092]) Wait longer for server to die in KillServer? (takes about 5 seconds on my box). (refs #3)

[chris]

(In [1094]) Reinstate ifdefs around code that should be disabled on Win32 (fake fork() seems to be a bad idea).

Comment spelling fixes. (refs #3)

[chris]

(In [1095]) Reinstate #ifdefs on Win32. (refs #3)

[chris]

(In [1096]) Catch exceptions from BackupStoreDaemon::Run and log them without killing the server process, on platforms where forking is disabled (Win32). (refs #3)

[martin]

• [923] ok
• [927] ok
• [928] ok
• [930] ok
• [931] ok
• [934] ok
• [935] ok
• [936] ok
• [937] ok
• [938] ok
• [940] ok
• [941] ok
• [942] ok
• [943] ok
• [944] ok
• [945] ok
• [946] ok
• [947] ok
• [948] ok
• [949] ok
• [950] ok
• [951] ok
• [952] ok
• [953] ok
• [954] ok
• [955] ok
• [956] ok
• [957] and [964] ok
• [958] and [963] ok
• [959] ok
• [960] ok
• [961] ok
• [962] ok
• [965] ok
• [966] ok
• [968] ok

[chris]

(In [1113]) Remove double initialisation (now done in infrastructure/buildenv-testmain-template.cpp) (refs #3)

[chris]

(In [1114]) Remove double initialisation of winsock library. (refs #3)

[chris]

Thanks Martin!

Merged [923] [927] [928] [930] [931] [934] [935] [936] [937] [938] [940] [941] [942] [943] [944] [945] [946] [947] [948] [949] [950] [951] [952] [953] [954] [955] [956] [957] [964] [958] [963] [959] [960] [961] [962] [965] [966] [968] to trunk.

Still outstanding: [840] [995] [1013] [1014] [1015] [1016] [1017] [1022] [1027] [1029] [1030] [1031] [1032] [1035] [1036] [1037] [1049] [1050]+[1051] [1052] [1053] [1054] [1055] [1058] [1059] [1062] [1064] [1070] [1071] [1072] [1073] [1075] [1076] [1077] [1078] [1079] [1080] [1081] [1082] [1083] [1084] [1085] [1086] [1087] [1088] [1089] [1090] [1091] [1092] [1094] [1095] [1096] [1113] [1114].

[chris]

(In [1143]) Fix compile warnings (refs #3)

[chris]

(In [1144]) Fix scoping to ensure that objects which allocate memory via the standard libraries, free it before the memleak tests, to avoid test failures (refs #3)

[chris]

(In [1146]) Force glibc to use new/delete to allocate memory and disable its internal pools for the unit tests, to make memory leak detection work (refs #3)

[chris]

(In [1150]) Fix memory leak when TLSContext is reinitialised (refs #3)

[chris]

(In [1151]) Fixed typo (refs #3)

[chris]

(In [1152]) Declare MEMLEAKFINDER_INIT and MEMLEAKFINDER_NO_LEAKS macros which reference function and class in DebugMemLeakFinder? only in debug mode (refs #3)

[chris]

(In [1153]) Free backtrace strings even in debug mode by suppressing warnings from DebugMemLeakFinder?, to avoid a memory leak (refs #3)

[chris]

(In [1155]) openlog inside MAINHELPER block, to ensure that any memory leaks will be caught (refs #3)

[chris]

(In [1156]) Setup MAINHELPER as early as possible, and clear it as late as possible, for consistency (refs #3)

[chris]

(In [1159]) * Track memory leaks in allocations via the standard libraries, and

avoid malloc/delete mismatches, by overriding standard new operator.

• Added another global enable flag to memleak finder, which is used to mark the end of static allocations and the start of dynamic code, since the memory leak detection is done before cleanup of static objects.
• Added a public guard class, to allow safe scoped disabling of memory leak detection.
• Added InternalAllocGuard? to protect against recursive loops when allocating memory inside the memory leak checker. (refs #3)

[chris]

(In [1160]) Fix memory leak false alarms caused by modifying static objects (refs #3)

[chris]

(In [1161]) Initialise memleak finder at the start of every program that uses MAINHELPER (all except unit tests). (refs #3)

[chris]

(In [1163]) Fixed control reaching end of non-void functions (refs #3)

[chris]

(In [1164]) Initialise memory leak finder in all unit tests (refs #3)

[chris]

(In [1167]) Revert [1096] as it causes infinite loops if the listening socket can't be opened (refs #3)

[chris]

(In [1168]) Properly revert [1096] (refs #3)

[chris]

(In [1169]) Don't try to write to the interprocess socket if it's not open (refs #3)

[martin]

• [840] ok
• [995] ok
• [1013] ok
• [1014] incorrect. otherlibs is the 5th argument
• [1015] ok
• [1016] ok, but why would you ever not want O_CREAT on an invisible temp file? Similarly, what's the idea of it defaulting to O_RDONLY?
• [1017] ok
• [1022] ok

[martin]

• [1027] ok
• [1029] ok
• [1030] ok
• [1031] ok (should be part of same changeset as 1027)
• [1032] ok
• [1035] ok
• [1036] ok
• [1037] what does this change do?
• [1049] ok, but error message at line 976 needs updating (or at least an improved explanation, I don't understand what it is trying to say)

[martin]

• [1050] + [1051] ok
• [1052] ok
• [1053] ok
• [1054] ok
• [1055] ok
• [1058] ok - this case insensitive stuff should probably be more generic. I believe HFS+ on Mac OS X is also case insensitive so this and other related changesets should work there too
• [1059] ok
• [1062] ok
• [1064] ok
• [1070] ok
• [1071] ok
• [1072] I'd like to know the underlying issue here, it's not clear to me why this is the right fix
• [1073] ok
• [1075] ok
• [1076] why are you using TRUE instead of true?
• [1077] ok
• [1078] I presume it works, so ok to merge. style comments: in the regexp you could have used (....){5} instead of having five of them. Not sure why you couldn't just ..\..\..\..\.. instead. Also why is require using \ but open using /?
• [1079] macros are nasty, prefer inline fns. Why undefining fstat only in these files? (needs explanation I think)
• [1080] ok

[martin]

• [1076] ok

[chris]

(In [1171]) Use gettimeofday() to increase accuracy of GetCurrentBoxTime?() on platforms which support it. Fixes busy waits for 1 second in backup client when time for next backup is not on a 1 second boundary (which it never is). (refs #3)

[chris]

(In [1176]) Moved intercept code to a library module to allow it to be used by test/bbackupd as well (refs #3)

[chris]

(In [1177]) * Try to kill any daemons left over from previous tests before running

new tests.

• Try to kill any remaining daemons at the end of the test (refs #3)

[chris]

(In [1178]) Catch any exceptions while handling a connection and report to user rather than terminating. Useful for non-forking servers like bbstored on Windows. (refs #3)

[chris]

(In [1181]) Added a test for diff termination if MaximumDiffingTime? is exceeded (refs #3, refs #9)

[chris]

(In [1184]) Replace old-style setitimers for KeepAliveTime and MaximumDiffingTime? with new Timer objects. (refs #3, refs #9)

[chris]

(In [1185]) Search for dlfcn.h and dlsym() (needed for new intercept code) (refs #3, refs #9)

[chris]

(In [1186]) Pass any command-line parameters from t-gdb to gdb (refs #3)

[chris]

(In [1187]) Added a header file for including in test/bbackupd/testbbackupd.cpp and other modules which might need intercepts in future.

Added opendir/readdir and lstat hook capability.

(refs #3, refs #9)

[chris]

(In [1188]) Document that restore -d can be used to restore deleted files in any directory, deleted or not. (refs #3)

[chris]

(In [1189]) Log failure to rename ID map files. (refs #3)

[chris]

(In [1190]) Added tests for keepalives while scanning large directories. (refs #3, refs #9)

[chris]

(In [1191]) Moved KeepAlive? timer to BackupClientContext? object.

Made timeout initialisation non-static, and a property of the context object. (perhaps should be in rParams, I know).

(refs #3, refs #9)

[chris]

(In [1192]) Send keepalives when needed while scanning large directories (refs #3, refs #9)

[chris]

(In [1193]) Make the timer test reliable by using nanosleep() instead of sleep(), since sleep() may use signals and interfere with SIGALRM, and also cannot be resumed if interrupted by a signal. (refs #3, refs #9).

[chris]

(In [1195]) Compile fix: include MemLeakFinder?.h even on release builds (refs #3)

[chris]

(In [1196]) Compile fix: include std::string definition (refs #3)

[chris]

Hi Martin,

Regarding [1014], I tried to put "-lpcre" in otherlibs, and the compile check works, but the autoconf macro doesn't add "-lpcre" to CXXFLAGS, so actual compiles fail.

I wasn't sure what to do about that, but adding "-lpcre" to the first argument does update CFLAGS correctly. Can you think of a better way to do this?

Cheers, Chris.

[chris]

Merged [840].

[chris]

(In [1198]) Fix more deadlocks by minimising the amount of stuff that the signal handler does. (refs #3, refs #9)

[chris]

(In [1222]) Improved debugging when a path to a location doesn't exist or can't be opened. (refs #3)

[chris]

(In [1223]) Added implementation of getopt from BSD (refs #3) (check license!)

[chris]

Merged: [995] [1013] [1015] [1017] [1022] [1027+1031] [1029] [1030] [1032] [1035] [1036] [1050]+[1051] [1052] [1054] [1058] [1070] [1071] [1073] [1076] [1077] [1078] [1080]

Waiting on Martin: [1014] [1049] [1079]

Waiting on me: [1016] [1072]

Waiting on dependencies:

• [1055], on [1016], should be applied at the same time
• [1059], on [1075] (ok) and [1087] (not yet reviewed)
• [1062], on [1016]

Already applied (sorry): [1037] [1053] [1064]

Regarding [1037], this is a compile fix, should have been part of [900].

Regarding [1049], sorry, I don't see what's wrong with the error message on line 976? It makes sense to me.

Regarding [1079], RaidFile? uses a lot of POSIX functions which are not emulated by our library, and which work with integer file handles rather than HANDLEs. Unfortunately, because of this, they are not compatible with our emulated fstat. It's easier to undefine our emulated fstat, and use the system fstat() instead, rather than implement emulations of all the other POSIX functions which RaidFile? uses.

Still outstanding: [1081] [1082] [1083] [1084] [1085] [1086] [1087] [1088] [1089] [1090] [1091] [1092] [1094] [1095] [1113] [1114] [1143] [1144] [1146] [1150] [1151] [1152] [1153] [1155] [1156] [1159]+[1195] [1160] [1161] [1164] [1169] [1171] [1176] [1177] [1178] [1181] [1184] [1185] [1186] [1187] [1188] [1189] [1190] [1191] [1192] [1193] [1196] [1198] [1222] [1223]

Rejected: [1096]

[chris]

Waiting on dependencies (reformatting):

• [1055], on [1016], should be applied at the same time
• [1059], on [1075] (ok) and [1087] (not yet reviewed)
• [1062], on [1016]

[chris]

Please review [1224].

[chris]

(In [1225])

• Rename Loggers class to Logging, which looks nicer.
• Fix type of "line" argument to loggers to int rather than std::string to match LINE
• Define necessary static objects
• Remove global condition on logging for now

(refs #3)

[chris]

(In [1226]) Change all logging to use the logging framework, as an example. (refs #3)

[chris]

(In [1227]) Add missing include of <string> header (refs #3)

[chris]

(In [1228]) Don't redefine the built-in operators new and delete unnecessarily (refs #3)

[chris]

(In [1229]) Compile fix for platforms without intercept capability (refs #3)

[chris]

(In [1230]) Partially revert [1229]

(In [1231]) Visual Studio 2005 compile fixes, thanks to Gary. Enable (and require) PCRE when building with Visual Studio 2005.

[chris]

(In [1232]) Updated docs for building MinGW builds on Linux (refs #3)

[chris]

(In [1234]) Added a new config option, LogAllFileAccess, which will log access to every file and scanning every directory. The current implementation is taken straight from the Boxi branch. To be extended shortly. (refs #3)

[chris]

(In [1235]) Added logging of directory scans and excluded files. (refs #3)

[chris]

(In [1236]) Always start by logging everything to syslog, even when running on the console.

Don't closelog() here, let Logging do that for us. (refs #3)

[chris]

(In [1237]) Cosmetic spacing fixes (refs #3)

[chris]

(In [1238]) Convert Daemon class to new logging framework (refs #3)

[chris]

(In [1239]) Fix spelling (refs #3)

[chris]

(In [1240]) Ignore symlink permissions on Darwin, where they can't be set properly after symlink creation (refs #3)

[chris]

(In [1241]) Rename makedistribution.pl to makedistribution.pl.in.

Auto-generate makedistribution.pl in configure, to replace @PERL@ in shebang line.

Process spec files as text files, to allow removing private parts. (refs #3)

[chris]

(In [1242]) Support building from an unofficial tarball (from svn) by changing %{distribution_dir} at the top (automatically in distributions made by infrastructure/makedistribution.pl)

Write our RPM version number into VERSION.txt and hence compile it in (refs #3)

[chris]

(In [1244]) Don't die on unreadable files (refs #3)

[chris]

(In [1245]) Be nice to rpm that doesn't like empty macros (refs #3)

[chris]

(In [1246]) Don't do things with essential side effects inside ASSERT() macros (refs #3, refs #9)

[chris]

(In [1247]) Fix building from distribution tarballs again (refs #3)

[chris]

(In [1248]) Add missing files to manifest, required by configure (refs #3)

[chris]

(In [1249]) Spacing (refs #3)

[chris]

(In [1250]) Detect reparse points and change the device number (refs #3)

[chris]

(In [1251]) Skip mount points and NTFS reparse points inside a location, even if not excluded (refs #3)

[chris]

(In [1252]) Add a new logging level, NOTICE, between INFO and WARNING (justification: we need two levels of output for LogAllFileAccess, neither of which are warnings, one is very verbose, but must not be compiled out like TRACE).

Make Loggers default to logging everything.

Make the global log level filter work. (refs #3)

[chris]

(In [1253]) Log important output at NOTICE level, instead of INFO (refs #3)

[chris]

(In [1254]) Allow Loggers to log everything, as is their wont. Master level will be set in Daemon, patch to follow. (refs #3)

[chris]

(In [1255]) Throw CommonException? AccessDenied? if we didn't get access to the file, for better error description (refs #3)

[chris]

(In [1256]) Add option parsing with getopt()

Add "-D" flag as SINGLEPROCESS equivalent

Add "-q" and "-v" options to control master logging level

Log fatal errors as FATAL rather than ERROR

Log daemon start and stop as NOTICE rather than INFO (refs #3)

[chris]

(In [1257]) Log AccessDenied? errors in a more concise form than general exceptions (refs #3)

[chris]

(In [1258]) Cosmetic whitespace fix (refs #3)

[chris]

(In [1259]) Compile fix for RHEL4 ([NICK]) (refs #3)

[chris]

(In [1260]) Check for exclude entries that end in a path separator, and log a warning (refs #3)

[chris]

(In [1261]) Automatically initialise logging at startup, with a static object, to log to console and syslog.

All logging objects automatically register themselves with the global logging system. (refs #3)

[chris]

(In [1262]) Allow suppressing warnings about unsupported file types by excluding those files by name, using ExcludeFile? (refs #3)

[chris]

(In [1263]) Added a BufferedStream? class that can be wrapped around an IOStream to improve read performance when many small reads will be performed, e.g. while reading directories and during housekeeping. (refs #3)

[chris]

(In [1264]) Use BufferedStream? to speed up housekeeping by about ten times. (refs #3)

[chris]

(In [1265]) Moved MakeFullPath? into its own library file so that we can share it (BackupQueries? needs it too) (refs #3)

[chris]

(In [1266]) When comparing, don't recurse into excluded directories, as their contents will not be on the store.

Fix display of filenames during compare when backing up the root directory (double slash) (refs #3)

[chris]

(In [1267]) Buffer directory reads (refs #3)

[chris]

(In [1268]) Buffer store file integrity checks (refs #3)

[chris]

(In [1269]) Improve error message when server fails to start (refs #3)

[chris]

(In [1270]) Stop locations from being deleted from the store just because their local directory can't be found, as suggested by Gary (refs #3)

[chris]

(In [1272]) Catch errors during restore (refs #3)

[chris]

(In [1273]) Check whether the restore target parent directory exists before trying to create the target directory or save restore info in the parent (refs #3)

[chris]

(In [1274]) Make BackupClientRestoreDir? return a result code compatible with BackupClientRestore? (refs #3)

[chris]

(In [1275]) Spelling fix (refs #3)

[chris]

(In [1276]) Don't throw an exception if we fail to delete a file in the way of restore, just log the error and return an error code (refs #3)

[chris]

(In [1277]) Catch exceptions while writing files, attributes, resume info and checking file existence (refs #3)

[chris]

(In [1278]) Log failure to delete files and directories (refs #3)

[chris]

Please review [1271] and [1279].

[chris]

(In [1306]) Win32 compile fix (refs #3)

[chris]

(In [1352]) Simplify check for PERL

Define substitution TARGET_PERL and preprocessor PERL_EXECUTABLE to a native Perl (not Cygwin) on Win32, since Cygwin perl will not run inside bbackupd for unit tests (e.g. SyncAllowScript?).

(refs #3)

[chris]

(In [1353]) Update build instructions for Cygwin/Win32 (refs #3)

[chris]

(In [1355]) #include Test.h to fix compile (needed for TEST_THAT)

Improved error message

Improved comments

(refs #3, merges parts of [712], [1289] and [1337])

[chris]

(In [1356]) Fix make error when optional file doesn't exist (refs #3, merges [1098])

[chris]

(In [1357]) Declare that we have O_BINARY but not <sys/types.h> for MSVC (refs #3, merges [760] and [763])

[chris]

(In [1358]) Watch out for our leak tracking data being destroyed and don't crash when subsequent objects are destroyed. (refs #3, merges [1341])

[chris]

(In [1360]) Add logging in remaining places where Common OSFileOpenError can be thrown. (refs #3, merges part of [1099] and [1359])

[chris]

(In [1361]) Update to match new recommended installation of pcreposix.h as regex.h. (refs #3, merges [1281], reverting [1233])

[chris]

(In [1369]) Add logging in remaining places where Common OSFileOpenError can be thrown.

Fix compile warning (signed vs unsigned comparison)

(refs #3, merges part of [1099], and [1368])

[chris]

(In [1371]) Add logging in remaining places where Common OSFileOpenError can be thrown.

(refs #3, merges part of [1099], and [1370])

[chris]

(In [1372]) Win32 compile fix

(refs #3, merges [1306])

[chris]

(In [1375]) Count a difference when a local directory does not exist, but it does exist on the server (refs #3)

[chris]

(In [1376]) Catch exceptions during restore and report them, rather than aborting bbackupquery (refs #3)

[chris]

(In [1377]) Report an appropriate error if the target path of the restore operation is not found (refs #3, merges [514])

[chris]

(In [1378]) Report number of files and directories which could not be compared separately from compare failures, and report the appropriate return code (refs #3, merges [651])

[chris]

(In [1380]) Run housekeeping in idle time on Win32 (refs #3)

[chris]

(In [1383]) Check for d_ino member in struct dirent (refs #3, merges [1351])

[chris]

(In [1384]) Record the file and line of first test failure (refs #3, merges [593])

[chris]

(In [1386]) Check for return code 3 (unreadable files) instead of return code 2 (refs #3, depends on [1378])

[chris]

(In [1387]) Moved SendCommands?(), HUPServer(), KillServer?() to lib/server/ServerCommands.h. All of these use lib/server/WinNamedPipeStream on Win32, so they don't belong in lib/common.

Made LaunchServer?() work on Win32.

Added constants for paths to executables, for use in tests, removing the need for #ifdefs and clumsy DIRECTORY_SEPARATORs in paths.

Added terminate_bbackupd() and wait_for_operation() functions.

Update unit tests to #include "ServerControl?.h" if they need it.

(refs #3)

[chris]

(In [1388]) Report file and line of memory leak test failures (refs #3, merges [714])

[chris]

(In [1389]) Throw an assertion error if a NULL timer is added (refs #3, merges [1367])

[chris]

(In [1391]) Win32 compile fixes (no gettimeofday(), no signal()) (refs #3)

[chris]

(In [1392]) Code formatting (cosmetic) (refs #3, merges [1345])

[chris]

(In [1395]) Catch exceptions thrown by closing the WinNamedPipeStream? during shutdown and suppress them. (refs #3, merges [1284])

[chris]

(In [1396]) Always include process.h, as we don't know whether it was detected or not (we don't have access to lib/common/BoxConfig.h in lib/win32) (refs #3)

[chris]

(In [1397]) Our timer code only supports ITIMER_REAL (refs #3)

[chris]

(In [1398]) We can't use lib/common here, so we don't have ASSERT() (refs #3)

[chris]

(In [1399]) Expanded character set conversion API to allow arbitrary conversions (needed to handle command lines with international encodings) (refs #3, merges [1038])

[chris]

(In [1400]) Print localised error messages rather than error codes for all errors (refs #3, merges [1046])

[chris]

(In [1401]) Improve GetErrorMessage?() by including the error number/code in the message (helps debugging on foreign langauge versions of Windows) (refs #3, merges [1364])

[chris]

(In [1402]) Fix handling of O_EXCL to behave just like Unix, not abused to lock files.

Add a new constant which specifies that files are to be locked open.

(refs #3, merges [1288])

[chris]

(In [1403]) We don't have access to DIRECTORY_SEPARATOR_ASCHAR in lib/win32 (refs #3, merges [1362])

[chris]

(In [1404]) Typo fix (refs #3)

[chris]

(In [1405]) Fix two memory leaks and one buffer overflow in codepage conversion code. (refs #3, merges [1340])

[chris]

(In [1406]) First attempt to achieve a more logical order in this chaos: reordered all typedefs to be clearer and more readable (refs #3, merges [766])

[chris]

(In [1407]) Added d_type member to struct dirent, initialise with S_IFDIR or S_IFREG

MinGW compile fix

(refs #3, merges [775])

[chris]

(In [1408]) Compile fix for [1397] (refs #3)

[chris]

(In [1409]) Use #defines to replace POSIX functions with emulated ones on MinGW, like we do on MSVC. This allows us to #undef them when we really need to use the original platform function (if available).

Disable emulated fstat() in raidfile (and use the platform one) by undefining fstat, since it doesn't use the other emulated file handling functions, or need Unicode support, and it can't take a filehandle returned by open() (only openfile()).

(refs #3, merges [1045])

[chris]

(In [1410]) Remove #define BOX_VERSION since we now get it from BoxVersion?.h via BoxPlatform?.h when building with MSVC, and from the Makefiles when building with MinGW. (refs #3, merges part of [634])

[chris]

(In [1412]) Reorder for clarity

Expose GetErrorMessage?()

Improve comments

(refs #3, merges [1365])

[chris]

(In [1413]) Add new syslog level emulations (refs #3, merges remainder of [1299])

[chris]

(In [1414]) Group remaining set*id() and get*id() functions. Improve comments about why they are being retained. (refs #3, related to [634])

[chris]

(In [1419]) Remove #ifdefs, no longer required (refs #3, merges [1418])

[chris]

(In [1420]) Fixes for cross-compiling from Ubuntu Breezy (refs #3, merges [573])

[invisnet]

[1083] An OVERLAPPED struct needs to be cleared before every use when talking to pipes, not just the first time.

The pipe is in message mode; why would there be more data in the buffer than is being asked for? Use stream mode if this is supposed to be a generic class, or provide some way to select the mode.

memcpy() is not guaranteed to work with overlapping buffers (line 288). Use memmove() instead.

[chris]

(In [1421]) Simplify wait code (refs #3)

[chris]

(In [1422]) Improve output messages when waiting for daemon to start (refs #3)

[chris]

(In [1423]) Test expected behaviour for modifying a file without changing its modtime, both tracked and untracked (refs #3)

[chris]

(In [1424]) Flush stdout after writing to it (refs #3)

[chris]

(In [1425]) Use unique name for compare log (refs #3)

[chris]

(In [1426]) Test that bbackupd reports an error when the backup failed due to an exception (refs #3)

[chris]

(In [1427]) Flush stdout when writing to it (refs #3)

[chris]

(In [1428]) Add a new notification constant, NotifyEvent_BackupError, for use when an exception occurs during the backup.

Make bbackupd notify sysadmin when an exception occurs during the backup, using this error code, and the notification string "backup-error".

Change NotifyEventMAX to be one greater than the highest notification constant, makes code more maintainable.

(refs #3)

[chris]

(In [1429]) Log at trace level by default in debug builds (refs #3)

[chris]

(In [1435]) Move lib/common/ServerControl.h to lib/server where it belongs, since it uses server functions (WinNamedPipeStream? on win32) (refs #3)

[chris]

(In [1437]) Fix compilation error reported by Torsten Boob (refs #3)

[chris]

(In [1465]) Reinitialise the OVERLAPPED structure each time we start a new overlapped read. Thanks to Charles Lecklider for pointing this out. (refs #3, merges part of [1458])

[chris]

(In [1466]) Use memmove() for overlapping source and destination buffer. Thanks to Charles Lecklider for pointing this out. (refs #3, merges [1442])

[chris]

(In [1467]) Change named pipe from message to byte mode. Thanks to Charles Lecklider for pointing this out. (merges [1464])

[chris]

Hi Charles, please could you review again [1083]+[1465]+[1466]+[1467]? I hope this addresses all the issues that you rightly pointed out. Thanks again for reviewing!

[chris]

(In [1469]) Add emulated rename() with path conversion. (refs #3, merges [1436] and [1438])

[chris]

(In [1470]) Initialise logging framework and set our program name to Box Backup (bbstored) (refs #3, merges [1462])

[chris]

(In [1471]) Use logging framework to reduce noise for those who don't want it (refs #3, merges [1439] [1440] [1443])

[chris]

(In [1472]) Reduce default logging level in debug builds from TRACE back down to INFO, to reduce noise in tests. (refs #3, merges [1441])

[chris]

(In [1473]) Trivial code simplification. (refs #3, merges [1444])

[chris]

(In [1474]) Use logging framework to remove timer noise for those who don't want it. (refs #3, merges [1445])

[chris]

(In [1475]) Fix header include order. (refs #3, merges [1446])

[chris]

(In [1476]) Remove newlines from syslog() messages. (refs #3, merges [1447])

[chris]

(In [1477]) Initialise logging framework and set sensible default verbosity levels in bbackupquery (refs #3, merges [1449])

[chris]

(In [1478]) Add new exception codes for use in bbackupd, and possibly other clients. (refs #3, merges [1455])

[chris]

(In [1479]) Add autogen_ClientException.o to testbbackupd extra objects, because it links against bbackupd objects which need this. (refs #3, merges [1456])

[chris]

(In [1480]) Improve logging of pipe errors by including the error message. (refs #3, merges part of [1458])

[chris]

(In [1481]) iUse our new ClientException? codes for clock skew and archive problems. (refs #3, merges part of [1459])

[chris]

(In [1482]) Better handle a case where a force-sync command comes in immediately after (or during) a sync, i.e. less than MinimumFileAge? seconds after the last one. In this case, just move back the syncPeriodStart by 1 second. (refs #3, merges part if [1459])

[chris]

(In [1483]) Reformat long lines for readability. (refs #3, merges remainder of [1459])

[chris]

(In [1484]) Compile fix to [1448] (refs #3)

[chris]

(In [1486]) Move all command socket communications to the worker thread, to avoid deadlocks.

Use events, and a message list protected by a critical section, to pass messages between threads.

(refs #3)

[chris]

(In [1487]) Add a quiet mode to compare command, to make test output easier to read. (refs #3, merges [1460])

[chris]

(In [1488]) Initialise logging framework and set our program name to "Box Backup (bbstored)". (refs #3, merges [1462])

[chris]

(In [1489]) Use logging framework to remove timer noise for those who don't want it. (refs #3, merges [1448])

[chris]

(In [1490]) Merge new m4 from trunk. (refs #3, merges [1453])

[chris]

(In [1491]) Indent cleanup (refs #3)

[chris]

(In [1492]) Added precompiled output from message compiler, for those who don't have it, e.g. using MinGW instead of MSVC. (refs #3, merges [526])

[chris]

(In [1493]) Fix unix path to bbstored (refs #3)

[chris]

(In [1494]) Use #defined paths for applications to fix cross unix/win32 compatibility.

Fix memory leak.

Reformat comments.

(refs #3)

[chris]

(In [1498]) Fix spurious/suprious typos.

Use #defined paths for executables to remove win32/unix differences.

Don't check for bbstored memory leaks on win32.

Wrap long lines for readability.

(refs #3)

[chris]

(In [1499]) Use #defined paths for executables to remove win32/unix differences. (refs #3)

[chris]

(In [1501]) Close RaidFile? before committing, so that commit can work on win32.

Typo fix.

(refs #3)

[chris]

(In [1502]) Allow '/' as a path separator on all platforms, even Win32 (refs #3, merges [1496])

[chris]

(In [1504]) Win32 fixes (paths to executables, and don't try things that don't work on Windows, like sending HUP signals) (refs #3)

[invisnet]

[1486] I don't know if I can do this, but I'm going to try :-)

I'd like to veto this change. You seem to have done a partial re-implementation of a completion port - see CreateIoCompletionPort? and friends.

[chris]

(In [1505]) Only include headers on systems which have them (refs #3)

[chris]

(In [1506]) Test for restoring to a directory whose parent does not exist (refs #3, merges [517])

[chris]

(In [1507]) Disable symlink tests on win32. (refs #3, merges part of [623])

[chris]

(In [1508]) Disable interrupted restore test on Win32 (uses fork) (refs #3, merges part of [623])

[chris]

(In [1510]) Use the old way to unzip files on win32/cygwin, since the pipe method doesn't seem to work here. (refs #3, merges part of [623])

[chris]

(In [1511]) Check return value using the macro, because ::system works differently on Windows than on Unix (refs #3, merges part of [623])

[chris]

(In [1520]) Add a warning that comparing locations with a final directory separator may fail (Stefan Tauner) (refs #3, merges [1512])

[chris]

(In [1522]) Don't log errors or throw exceptions when we get ERROR_NO_DATA, which just means that the pipe is being closed. Treat it as a normal remote close (EOF) instead.

Don't log an error if DisconnectNamedPipe? tells us that the remote end already closed the pipe (ERROR_PIPE_NOT_CONNECTED).

Treat ERR_PIPE_NOT_CONNECTED during pipe reads as EOF as well.

Improve logging of pipe errors by including the error message.

(refs #3, merges [1458] and [1463])

[chris]

(In [1524]) Rearrange #includes in alphabetical order. (refs #3)

[chris]

(In [1525]) Compile fix, always include Test.h (refs #3)

[chris]

(In [1527]) Use #defined constants rather than hard-coded paths for launching programs. (refs #3)

[chris]

(In [1531]) Comment out some tests that don't work on Win32 (refs #3)

[chris]

(In [1533]) Add utility functions for running different types of syncs, set Windows file time (enhanced utimes()), and finding a named entry in a directory on the store. (refs #3)

[chris]

(In [1534]) Don't fill in struct dirent.d_ino unless we've detected that it's present on this platform. (refs #3)

[chris]

(In [1538]) Work around lack of pipe support in Win32 system() command (refs #3)

[chris]

(In [1539]) Enhance test for bbackupd pausing on change of client store marker (refs #3)

[chris]

(In [1540]) Compile fix, comment out set_file_time helper function on non-Win32 platforms (refs #3)

[chris]

(In [1541]) Test fix: create TestDir1 with a readable mode (refs #3)

[chris]

(In [1543]) Keep bbackupd pid in a global variable instead of a local one.

Stop bbackupd nicely with terminate_bbackupd() instead of killing it, so that we get a memory leak report on Win32.

Stop bbackupd and bbstored at end of test if any tests fail.

(refs #3)

[chris]

(In [1544]) Test that filenames containing non-ASCII (8-bit) characters can be backed up and restored, and directories can be changed into and out of, on win32. (refs #3)

[chris]

(In [1545]) Detect whether getopt.h is present on system. (refs #3)

[chris]

(In [1546]) Pass all arguments from test scripts to test executable with proper quoting (refs #3)

[chris]

(In [1547]) Declare global std::string variables that hold extra arguments for executables run in tests, to be implemented in a follow-up patch.

ConvertPaths?, LaunchServer? and RunCommand? all take std::string references instead of char*.

Move safe_sleep from test/common to lib/server/Test.h so that all tests can use it.

(refs #3)

[chris]

(In [1548]) Use logging framework to log exceptions. (refs #3)

[chris]

(In [1549]) Use logging framework to log stack traces from exceptions. (refs #3)

[chris]

(In [1550]) Log event name instead of numeric code. (refs #3)

[chris]

(In [1551]) Add options to log timestamps, and a custom tag, with each message to the console, e.g.:

14:53:17 [bbackupd] Finished scan of local files

(refs #3)

[chris]

(In [1552]) Add -V option, which sets maximum verbosity in one go.

Add -T option, which adds timestamps to console logs.

Add -t option, which enabled and sets the tag used on console logs.

Add -k option, which keeps console open after forking.

(refs #3)

[chris]

(In [1553]) Add --bbackupd-args and --bbstored-args options, which may be used to pass additional command-line parameters to bbackupd and bbstored in some tests.

Add -t and -T options, which add tags and timestamps to log messages using the logging framework inside the tests themselves.

Now you can do things like:

./t -Tt test \

--bbackupd-args="-VTkt bbackupd" \ --bbstored-args="-VTkt bbstored"

and you will get output from the test itself, the bbackupd and bbstored daemons, all interleaved, timestamped and tagged on the console.

This is very useful for debugging synchronisation problems, where interleaved output from bbackupd and bbstored in the system logs is not enough, because you want to see exactly how they interact with the test, or you don't have access to the system logs.

(refs #3)

[chris]

(In [1554]) Use additional arguments from command line options when starting bbackupd and bbstored. (refs #3)

[chris]

(In [1555]) Replace sleep() with safe_sleep() everywhere (refs #3)

[chris]

(In [1556]) Fix excessively verbose warnings while waiting for daemon to start. (refs #3)

[chris]

(In [1557]) Use TARGET_PERL instead of PERL in shebang lines (refs #3)

[chris]

(In [1558]) Add sync allow script to test config, to allow it to be tested.

Use TARGET_PERL to execute the test scripts. (refs #3)

[chris]

(In [1559]) Test that SyncAllowScript? is executed and can pause backup for the correct amount of time. (refs #3)

[chris]

(In [1563]) Display file and line where memory leak test failed. (refs #3)

[chris]

(In [1565]) Compile fix for Win32, where no localtime_r is available and localtime is thread safe. (refs #3)

[chris]

(In [1567]) Merge [1566] from chris/general:

Use Sleep() instead of nanosleep again on win32 (lost in merge [1562]).

Fix reference to pCommandLine which no longer exists after [1562].

Fix signed/unsigned comparison warning. (refs #3)

[chris]

(In [1570]) Fix accidental passing of object to printf() (refs #3)

[chris]

(In [1572]) Use LaunchServer? to remove the need for some #ifdefs.

Use directory separator constant instead of forward slash.

Add some blank lines for readability. (refs #3)

[chris]

(In [1575]) Use wait_for_sync_end() rather than calling bbackupquery ourselves (refs #3)

[chris]

(In [1577]) Ignore attempts to listen on a Unix socket on Win32, so that we can share config files for the unit tests (refs #3)

[chris]

(In [1578]) Use Cygwin chmod command-line tool to make the read-only file writable on Win32, instead of our own chmod() (which does nothing). (refs #3)

[chris]

(In [1579]) Use wait_for_sync_end() to synchronise the test more accurately with bbackupd, hopefully reducing random failures. (refs #3)

[chris]

(In [1581]) Check that Exclude and AlwaysInclude? configurations actually work as they should. (refs #3, merges [711])

[chris]

(In [1582]) Compare restored files as part of restore test (refs #3, merges part of [623])

[chris]

(In [1583]) Test that file attributes are backed up and compared correctly on Win32 (refs #3, merges part of 694])

[chris]

(In [1584]) Test that locked files behave as expected on Win32 (refs #3, merges parts of [649], [694], [1461])

[chris]

(In [1585]) Add a stream which provides a source of zero bytes of arbitrary size, useful for testing support for files over 2GB. (refs #3)

[chris]

(In [1586]) Fix inability to handle streams over 2GB properly. (refs #3)

[chris]

(In [1587]) Test that PartialReadStream? and ReadGatherStream? can handle streams over 2GB in size. (refs #3)

[chris]

(In [1588]) Keep a copy of the program name, in case the original goes away, which causes syslog() to write nonsense for the program name. (refs #3)

[chris]

(In [1589]) Add a stream which logs progress of reading data from another (child) stream, and estimated time of completion, useful for upload progress monitoring. (refs #3)

[chris]

(In [1590]) Use ReadLoggingStream? to log progress of file uploads. (refs #3)

[chris]

(In [1591]) Make Daemon remember whether we're supposed to run in a single process, or not. (refs #3)

[chris]

(In [1592]) Remove comments accidentally left in [1591]. (refs #3)

[chris]

(In [1593]) Don't fork to handle requests when running as a single process (very useful for debugging). (refs #3)

[chris]

(In [1594]) Improve log message for NotifyFileUploading?(). (refs #3)

[chris]

(In [1595]) Notify about full file uploads. (refs #3)

[chris]

(In [1596]) Fix formatting bugs. (refs #3)

[chris]

(In [1597]) Improve error messages when Getting a file fails with an exception. (refs #3)

[chris]

(In [1598]) Fix getting files with uncertain size (over 2GB) from the store. Failure to drain the stream will leave the EOF byte in it, which breaks further communications with the store over the same connection. (refs #2, refs #3)

[chris]

(In [1599]) Fix bbackupd choosing an invalid (too large) block size for large files (over 2GB) which will cause compare to fail: when rBlockSizeOut == BACKUP_FILE_MAX_BLOCK_SIZE we would have proceeded around the loop one more time and doubled the block size again. (refs #2, refs #3)

[chris]

(In [1600]) Improve error messages when compare throws an exception. (refs #3)

[chris]

(In [1601]) Remove newline from end of banner text, since logging framework will add it for us where necessary. (refs #3)

[chris]

(In [1602]) Use logging framework for banner. (refs #3)

[chris]

(In [1604]) Return an error code on failure of lcd command.

Return an error code if get command fails due to the file already existing.

(refs #3, merges [1516])

[chris]

(In [1605]) Fix rare memory leak. (refs #3, merges part of [1514])

[chris]

(In [1606]) Listen on an inet socket as well, to keep tests happy on Windows (refs #3, merges part of [582])

[chris]

(In [1607]) Revert to Unix paths (refs #3)

[chris]

(In [1608]) Test that bbackupd and bbstored are still running at various points during the test, and abort early if not. (refs #3)

[chris]

(In [1609]) Format messages within test to make them more obvious, and improve readability of test output. (refs #3)

[chris]

(In [1610]) Use quiet mode in compares, to reduce test output verbosity (refs #3)

[chris]

(In [1611]) Use helper function rather than calling bbackupctl directly. (refs #3)

[chris]

(In [1613]) Close updated files during continuous-update test, so that they can be backed up on Win32. (refs #3)

[chris]

(In [1614]) Remove misplaced hunk. (refs #3)

[chris]

(In [1615]) Call perl executable to execute external scripts, since shebang doesn't work on win32. (refs #3)

[chris]

(In [1616]) Gratuitous cosmetic reformatting, to make the test easier to read (refs #3)

[chris]

(In [1617]) Initialise mSingleProcess flag to false (oops!) (refs #3)

[chris]

(In [1618]) Make directories user-writable while restoring them, so that restoring read-only directories as a non-root user doesn't crash. Restore the correct permissions after restoring the directory's contents. (refs #3)

[chris]

(In [1619]) Make directories user-writable while restoring them, so that restoring read-only directories as a non-root user doesn't crash. Restore the correct permissions after restoring the directory's contents. (refs #3)

[chris]

(In [1620]) Remove unwanted blank line. (refs #3)

[chris]

(In [1621]) Remove assertion that we are at the end of the stream when decoding finishes. This is not true when decoding raw files on disk. (refs #3)

[chris]

(In [1622]) Report Restore_UnknownError properly during restore.

Report unknown result codes. (refs #3)

[chris]

(In [1623]) Read any remaining data from the encoded stream (such as EOF marker) before discarding it, to ensure that we don't break the protocol. (refs #2, refs #3)

[chris]

(In [1624]) Compile fix, thanks to Matt Brown (refs #3)

[chris]

(In [1625]) Fix running common tests in release mode, by not doing things that crash in release mode, but assert in debug mode. (refs #3)

[chris]

(In [1626]) Add missing #include <errno.h>, thanks to Matt Brown (refs #3)

[chris]

(In [1627]) Read-only restore test fix (refs #3)

[chris]

(In [1628]) Make testfiles writable before trying to delete them, otherwise read-only tests might not work (refs #3)

[chris]

(In [1629]) Slightly more progress information during tests (refs #3)

[chris]

(In [1646]) Fix emulated chdir to work with relative paths and with bbackupquery's "sh" command (which doesn't like UNC paths).

Fix error messages by removing surplus newline kindly added by Windows.

(refs #3, merges [1514] and [1569])

[chris]

(In [1648]) Gratuitous additional spaces (refs #3)

[chris]

(In [1649]) Remove old support for Cygwin.

Remove lib/win32 dependency on lib/server (circular).

Remove all dependencies on lib/win32.

Enable all modules and tests on Win32.

(refs #3)

[chris]

(In [1653]) Fix the license in the docs to reflect the actual license. (refs #3, merges [1652])

[chris]

(In [1699]) Include <string> to avoid compile failures on BSD, thanks James O'Gorman. (refs #3, merges [1698])

[chris]

(In [1713]) Make test/bbackupd intercept timer tests a little less strict, because object IDs can vary depending on the order that readdir() returns files in. (refs #3)

[chris]

(In [1715]) Add --with-bdb-headers and --with-bdb-lib flags to configure to fix Berkeley DB detection issues on FreeBSD. The original --with-bdb-dir flag is still present for systems that install BDB in a "standard" location. (merges [1714] from trunk by James O'Gorman, refs #3)

[chris]

(In [1719]) Remove --with-bdb-dir option, to avoid confusion and be consistent with the SSL flags. (merges [1717] by James O'Gorman, refs #3)

[chris]

(In [1720]) Hopefully work around buggy getopt implementation noted by TBP in  http://lists.warhead.org.uk/pipermail/boxbackup/2007-July/003614.html (refs #3)

[chris]

(In [1722]) Make char signed to fix getopt usage on ARM and PowerPC, thanks to TBP and Reinhard Tartler (refs #3, see  http://lists.warhead.org.uk/pipermail/boxbackup/2007-July/003614.html)

[chris]

(In [1724]) Use a signed int instead of char for getopt, thanks Martin. (refs #3)

[chris]

(In [1726]) Hopefully fix a bug reported by Tobias Balle-Petersen, where unused locations on the store would never be deleted because the timer would be reset on every backup pass (refs #3).

[chris]

(In [1732]) Add missing #include <errno.h>, thanks Tobias. (refs #3, merges [1694])

[chris]

(In [1734]) Hopefully reset getopt in a way that's compatible with BSD and GLIBC, thanks to Tobias Balle-Petersen for pointing out this bug. (refs #3, merges [1695])

[chris]

(In [1735]) Get it right this time. (refs #3, merges [1696])

[chris]

(In [1736]) Handle USE_SVN_VERSION in distribution VERSION files. (refs #3, merges [1702])

[chris]

(In [1737]) Use SVN version in distributions built from this branch. (refs #3, merges [1703])

[chris]

(In [1738]) Add lib/intercept to distribution manifest, to ensure that it gets distributed. (refs #3, merges [1704])

[chris]

(In [1739]) Handle USE_SVN_VERSION in the middle of a version string. (refs #3, merges [1705])

[chris]

(In [1740]) Replace version in VERSION.txt, if it contains USE_SVN_VERSION. (refs #3, merges [1706])

[chris]

(In [1741]) Add a new -F option for daemons, which runs in the foreground but still accepts multiple connections, unlike -D or SINGLEPROCESS. (refs #3, merges [1707])

[chris]

(In [1742]) Log reasons for failure to connect to a socket. (refs #3, merges [1708])

[chris]

(In [1743]) Make test/bbackupd intercept timer tests a little less strict, because object IDs can vary depending on the order that readdir() returns files in. (refs #3, merges [1713])

[chris]

(In [1744]) iWarn if an exception is thrown, with line number, as this shouldn't be happening on production systems anyway. (refs #3, merges [1727] and [1730])

[chris]

(In [1745]) Add more debugging checks and messages for location setup error reported by Pete Jalajas ( http://lists.warhead.org.uk/pipermail/boxbackup/2007-July/003668.html) (refs #3, merges [1728])

[chris]

(In [1746]) More helpful error messages when location setup failed. (refs #3, merges [1729])

[chris]

(In [1748]) Create logger object in the right place to avoid accidental triggering. (refs #3, merges [1747])

[chris]

(In [1749]) License update. (refs #3, merges [1654])