Merge Win32 branch

[chris]

The changes from the General branch need to be merged into trunk, or rejected.

Merge via the Merge branch for review and testing.

[chris]

Please review http://bbdev.fluffy.co.uk/trac/changeset/817

[chris]

Please review http://bbdev.fluffy.co.uk/trac/changeset/814

[chris]

Please review http://bbdev.fluffy.co.uk/trac/changeset/798

[chris]

Please review http://bbdev.fluffy.co.uk/trac/changeset/749

[chris]

Please review http://bbdev.fluffy.co.uk/trac/changeset/819

[chris]

Please review [821]

[chris]

Please review [825]

[chris]

Please review [827]

[chris]

Please review [829]

[chris]

Please review [831]

[chris]

Please review 832

[chris]

Please review [833]

[martin]

[817] ok to merge

[814] ok to merge

[798] ok to merge but lines 277 to 283 (catch BoxException?) are redundant because BoxException? is derived from std::exception.

[749] ok to merge

[chris]

Please review [835]

[chris]

Thanks Martin, I've merged 817, 814, 798 (with change) and 749.

[martin]

Ok to merge: [821] [825] [827] [829] [831] [832]

[martin]

Ok to merge: [833] [835]

Excepting [819] queried on -dev, up to date at this point.

[chris]

Thanks again Martin, please review [840].

[chris]

All merged to date except [819] and [840].

[chris]

Please review [860]

[chris]

Please review [861]

[chris]

(In [863]) * bin/bbstored/BackupStoreDaemon.h - Reinstate SendMessageToHousekeepingProcess?() on Win32, but make it do

nothing (refs #3)

[chris]

Please review [862] and [863]

[chris]

(In [864]) * configure.ac

Revert to trunk (refs #3)

[chris]

(In [867]) * docs/backup/win32_build_on_linux_using_mingw.txt (refs #3)

Updated documentation for building Win32 native builds on Linux

[chris]

(In [869]) * infrastructure/BoxPlatform.pm.in

Determine whether we are building for Windows or not, and export that information

Search for VERSION.txt in a few more places, for example when running Perl scripts like getversion.pl

(refs #3)

[chris]

(In [871]) * infrastructure/buildenv-testmain-template.cpp

Record the file and line of first test failure, and print them at the end of the test, useful for debugging when the first failure has scrolled off screen (refs #3)

[chris]

(In [873]) * infrastructure/makebuildenv.pl.in

Support multiple implicit dependencies

Add lib/win32 as an implicit dependency on Win32

Make lib/common depend on lib/win32 on Win32

Fix space between test file name platform executable extension, and between executable name and arguments, in test shell scripts

Add support for compiling resource files (.rc) on Win32

Fix dependencies to avoid unnecessary rebuilds when not using BSD make

Code cleanups

Fix typos

(refs #3)

[chris]

(In [875]) * lib/backupclient/BackupClientFileAttributes.cpp

Don't call geteuid() on Win32, since it's emulated, always returns 0, and I want to remove it entirely (refs #3)

[chris]

(In [877]) * lib/backupclient/BackupStoreFile.cpp

Close file before trying to apply attributes to it. Otherwise, when we close it the timestamp will be updated on Win32 (refs #3)

[chris]

(In [879]) * lib/backupclient/BackupStoreObjectDump.cpp

Fix format strings on Win32 (refs #3)

[chris]

Please review [881]

[chris]

(In [883]) Use DIRECTORY_SEPARATOR instead of assuming that it's a forward slash (refs #3)

[chris]

(In [885]) Disable intercept tests on Win32 as well

Define O_BINARY to 0 (zero) if our platform doesn't have it (all except Win32?) which enables us to reduce #ifdefs

(refs #3)

[chris]

(In [887]) Make FDGETLINE_BUFFER_SIZE big enough for one unicode character on Win32, otherwise reading from console can fail due to insufficient buffer size. (refs #3)

[chris]

(In [889]) Add O_BINARY to default flags, since most files opened this way should be opened in binary mode on Win32 (refs #3)

[chris]

(In [892]) Disable all calls to set*id() on Win32 (doesn't work) (refs #3)

[chris]

(In [894]) (refs #3)

Don't include headers that we don't have

Open RAID files in binary mode

[chris]

(In [896]) (refs #3)

Open files in binary mode (Win32)

Disable the lock failure block when we don't have any locking mechanism

Close and delete files before renaming over them on Win32. This breaks Ben's desired recovery semantics, so it's not done on other platforms, but Win32 requires it.

[martin]

Ok to merge: [860] [863] [881]

[861] is ok except for the unnecessary call to Close as mailed.

[862] is ok but now the file is being deleted twice (once inside the try/catch block and once after). Should remove the one outside the try/catch.

Still outstanding: [840]

[chris]

(In [898]) (refs #3)

Initialise Windows sockets automatically for all daemons on Win32

Write PID files on Win32

[chris]

(In [900]) (refs #3)

64-bit format fixes (Win32)

[chris]

(In [902]) (refs #3)

Added an OnIdle? method which can be overridden by subclasses for idle tasks. Used for housekeeping on Win32.

Avoid forking on Win32, and trying to clean up after children.

[chris]

Sorry, please review [865] instead of [864]

[chris]

(In [904]) (refs #3)

Define a suitable constant for invalid socket handles, for use in SocketStream?.cpp

[chris]

Please review [906]

[chris]

(In [908]) (refs #3)

Added support for non-blocking sockets on Win32 as well

[chris]

(In [910]) (refs #3)

Use strict and warnings for code safety.

Exit with status 2 on invalid usage.

Allow running multiple tests, comma separated.

Exit with status 2 if make fails for any test.

Exit with status 1 if any test fails.

[chris]

(In [912]) (refs #3)

Use truncation/sparse files to quickly create files with 200MB of zeroes

[chris]

(In [913]) (refs #3)

64-bit format string fixes for Win32

[chris]

(In [914]) (refs #3)

Use CopyFile? to copy on Windows, instead of shelling out to cp, which doesn't understand native paths.

[chris]

(In [915]) (refs #3)

Close open file handles on Win32

[chris]

(In [916]) (refs #3)

Completely disable symlink test on Win32

[chris]

(In [917]) (refs #3)

Cosmetic fixes to comments

[chris]

(In [918]) (refs #3)

This test takes a long time on Win32 (slow file access? VMware?), so:

• Disable verbose debug logging, which makes it even slower
• Print a warning before the slow test starts
• Increase timeout

[chris]

(In [920]) (refs #3)

Close before unlink

Check that unlink succeeds

[chris]

(In [921]) (refs #3)

We cannot open multiple connections to the server on Win32, so work around.

[chris]

(In [922]) (refs #3)

Use correct directory separator everywhere

[chris]

(In [923]) (refs #3)

Disable symlink tests on Win32

[chris]

(In [927]) (refs #3)

Convert UNIX paths to native on Win32 (avoids #ifdefs in tests)

[chris]

(In [928]) (refs #3)

Disable tests for memory leaks after killing bbstored on Win32, since it dies forcibly without a chance to write the memory leak file.

Initialise Windows sockets on Win32

Add tests for multiply open files on Win32

[chris]

(In [930]) (refs #3)

Remove newlines from output to avoid failure to recognise it on Win32

[chris]

(In [931]) (refs #3)

Added a RunCommand?() function which converts UNIX paths to native before calling ::system() on Win32.

[chris]

(In [932]) (refs #3)

Call RunCommand?() instead of ::system to handle native paths better

[chris]

(In [934]) (refs #3)

Initialise Windows sockets on Win32

[chris]

(In [935]) (refs #3)

Fix format strings on Win32

[chris]

(In [936]) (refs #3)

Cannot signal bbstored to housekeep immediately on Win32, so just wait for it to happen.

[chris]

(In [937]) (refs #3)

Don't check for memory leaks on Win32, since the process is force killed and doesn't have a chance to write the log file.

[chris]

(In [938]) (refs #3)

Use RunCommand? instead of ::system to convert UNIX paths to native

[chris]

(In [940]) (refs #3)

Sleep without nanosleep() on Win32

[chris]

(In [941]) (refs #3)

Automatically initialise Windows sockets for all tests

[chris]

(In [942]) (refs #3)

Reorganise typedefs for clarity

[chris]

(In [943]) (refs #3)

Remove dependency on Box.h and hence on lib/common

[chris]

(In [944]) (refs #3)

Add message definitions and resource files

[chris]

(In [945]) (refs #3)

Include message resource definitions

[chris]

(In [946]) (refs #3)

Add BSD-licensed getopt.h to replace our own version

[chris]

(In [947]) (refs #3)

Use our own ConvertUtf?8ToWideString for simplicity

[chris]

(In [948]) (refs #3)

Convert UNIX to native paths

[chris]

(In [949]) (refs #3)

Return INVALID_HANDLE_VALUE instead of NULL from openfile() on failure

[chris]

(In [950]) (refs #3)

Use INVALID_HANDLE_VALUE instead of NULL for invalid handles on Win32

[chris]

(In [951]) (refs #3)

Improved handling of UNIX flags to openfile(), allows multiple opens

[chris]

(In [952]) (refs #3)

Improved error handling in emu_fstat

[chris]

(In [953]) (refs #3)

Always fill in st_uid, st_gid and st_nlink

Return zero size for directories

Improved emulation of Unix file modes (helps when restoring Windows files on Unix)

[chris]

(In [954]) (refs #3)

Pass requested open flags to OpenFileByNameUtf?8

[chris]

(In [955]) (refs #3)

Request READ_CONTROL when standard open fails, rather than no flags

[chris]

(In [956]) (refs #3)

Treat PATH_NOT_FOUND just like FILE_NOT_FOUND in Unix land (errno = ENOENT)

[chris]

(In [957]) (refs #3)

Added emulated utimes, chmod, readv and writev

Added useful utility functions ConvertFileTimeToTime?_t and ConvertTime?_tToFileTime

[chris]

(In [958]) (refs #3)

Removed last vestiges of old getopt()

[chris]

(In [959]) (refs #3)

Improved poll() emulation to handle multiple fds

[chris]

(In [960]) (refs #3)

Fixed looking up message source in syslog()

Flush stdout after writing to it

Allow syslog() to log to console even when openlog() has not been called

[chris]

(In [961]) (refs #3)

Emulated chdir, mkdir and unlink should handle file names in UTF-8 as well

[chris]

(In [962]) (refs #3)

Emulate readdir's d_type field

[chris]

(In [963]) (refs #3)

Remove more vestiges of old getopt

[chris]

(In [964]) (refs #3)

Added prototypes for new emulated functions emu_utimes, readv and writev

[chris]

(In [965]) (refs #3)

Remove inline ConvertFileTimeToTime?_t, add prototypes for new versions in emu.cpp

[chris]

(In [966]) (refs #3)

Changed WideSize? from int to size_t

Don't allow space for null terminator that will not be added

[chris]

(In [968]) (refs #3)

Fixed typo

[chris]

Hi Martin,

Currently outstanding as I understand it:

• [819] waiting for me to implement InvisibleTempFileStream?
• [840] waiting for Ben to be satisfied that there are sufficient tests of this functionality
• [843] waiting for me to rewrite to re-enable locking on Win32
• [860] approved, already applied?
• [861] waiting on me to remove #ifdefs
• [862] needs InvisibleTempFileStream?
• [863] approved, already applied?
• [865]
• [867]
• [869]
• [871]
• [873]
• [875] rejected
• [877]
• [879]
• [881] approved, already applied?
• [883]
• [885]
• [887]
• [889]
• [892] rejected, but the comment spelling fix should be applied
• [894]
• [896] rejected, needs a rewrite as Ben suggested
• [898]
• [900]
• [902]
• [904]
• [906]
• [908]
• [910]
• [912]
• [913]
• [914]
• [915]
• [916]
• [917]
• [918]
• [920]
• [921]
• [922] rejected DIRECTORY_SEPARATOR, but keep the f.Close();
• [923]
• [927]
• [928]
• [930]
• [931]
• [934]
• [935]
• [936]
• [937]
• [938]
• [940]
• [941]
• [942]
• [943]
• [944]
• [945]
• [946]
• [947]
• [948]
• [949]
• [950]
• [951]
• [952]
• [953]
• [954]
• [955]
• [956]
• [957] and [964]
• [958] and [963]
• [959]
• [960]
• [961]
• [962]
• [965]
• [966]
• [968]

[martin]

[865] ok [867] ok [869] please use $OSNAME (probably need 'use English' too), $^{O is horrible, go ahead & merge when you've made that change [871] ok, or you could give -l 10000 to xterm :-) (maybe not so easy in mswin) [873] ok [877] ok [879] ok [883] ok [885] ok [887] ok [889] ok [894] ok}

[martin]

(Just reformatting)

• [865] ok
• [867] ok
• [869] please use $OSNAME (probably need 'use English' too), $^O is horrible, go ahead & merge when you've made that change
• [871] ok, or you could give -l 10000 to xterm :-) (maybe not so easy in mswin)
• [873] ok
• [877] ok
• [879] ok
• [883] ok
• [885] ok
• [887] ok
• [889] ok
• [894] ok

[chris]

(In [981]) * Close file before trying to apply attributes to it. Otherwise, when we close it the timestamp will be updated on Win32 (refs #3)

[chris]

Thanks Martin! Please ignore [981].

Merged [865] [867] [869] [871] [873] [877] [879] [883] [885] [887] [889] [894] into trunk.

[chris]

Merged [861] [863] [881] to trunk.

[martin]

• [898] ok
• [900] ok
• [902] ok
• [904] & [906] ok (these should have been one patch, both implementing the same change)
• [908] ok, though ifdefs such as this one would be better based on the feature (ie. presence of fcntl/O_NONBLOCK) rather than the platform. We'll need to revisit these later.
• [910] ok
• [912] ok
• [913] ok
• [914] ok
• [915] ok
• [916] ok
• [917] ok
• [918] to discuss
• [920] ok
• [921] ok

[chris]

(In [995]) * Added support for non-blocking sockets on Win32 as well (TODO revisit #ifdefs: refs #3)

[chris]

Thanks Martin!

Merged [898] [900] [902] [904] [906] [908] [910] [912] [913] [914] [915] [916] [917] [920] [921] to trunk.

[chris]

Please review [1013] [1014] [1015] [1016] for merge.

[chris]

Please review [1017] for merge.

[chris]

(In [1019]) * Comment grammar fix (refs #3)

[chris]

Sorry, please ignore [1019].

[chris]

(In [1022]) * Use readdir's d_type field to determine type of directory entry without

statting the entry, which allows users to exclude unreadable entries to suppress warnings about them on Win32 (refs #3)

[chris]

(In [1026]) * Apply Box coding standards (refs #3)

[chris]

(In [1027]) * Replace global daemon object with a pointer, to allow deletion and

clean up reported memory leaks

• No need to initialise Winsock here, now that lib/server/Daemon does it for us
• Initialise logging properly (refs #3)

[chris]

(In [1029]) * Apply Box coding standards (refs #3, replaces [1026])

[chris]

(In [1030]) * Catch invalid output from sync allow script and tell the user what it

was, to help them debug the problem (refs #3)

[chris]

(In [1031]) * Replace global daemon object with a pointer, to allow deletion and

clean up reported memory leaks (refs #3, combine with [1027])

[chris]

(In [1032]) * Operator precedence fix (refs #3)

[chris]

(In [1035]) * Fix compile error caused by conflicting MSVC macros (refs #3)

[chris]

(In [1036]) * Add option to bbackupquery list command to show times in UTC or local time

(refs #3)

[chris]

(In [1037]) Use BoxPlatform? to see if we are building on Windows (refs #3)

[chris]

(In [1049]) Allow getting an object with a path, not just from the current directory (refs #3)

[chris]

(In [1050]) Convert command-line arguments from the system locale/character set to the console character set (code page), so they they can be converted from console to UTF-8 (yuck).

Don't try to read from stdin or change its code page when it's not open (invalid file handle) (refs #3)

[chris]

(In [1051]) Convert command-line arguments from the system locale/character set to the console character set. (refs #3, combine with [1050])