From: Robert Nelson Date: Sun, 5 Nov 2006 12:34:59 +0000 (+0000) Subject: Fixed problems with encryption when combined with compression or sparse files. Unfor... X-Git-Tag: Release-2.0.0~318 X-Git-Url: https://git.sur5r.net/?a=commitdiff_plain;h=242be8cb2c8f43fbfd6e6ca261c6f8ae4da1a59a;p=bacula%2Fbacula Fixed problems with encryption when combined with compression or sparse files. Unfortunately because of this bug all previous encrypted backups can't be restored. Fixed problem in bfgets with Windows and Mac end of lines. Fixed bug in Windows version of close_bpipe() that caused programs like bsmtp() to hang waiting for the pipe to be closed. Fixed binary file type problems in open_bpipe(). Fixed a number of bugs in the Windows version of the mysql DB scripts. git-svn-id: https://bacula.svn.sourceforge.net/svnroot/bacula/trunk@3599 91ce42f0-d328-0410-95d8-f526ca767f89 --- diff --git a/bacula/ReleaseNotes b/bacula/ReleaseNotes index e1282c023f..4892fcfc62 100644 --- a/bacula/ReleaseNotes +++ b/bacula/ReleaseNotes @@ -69,18 +69,18 @@ Version 1.39.26 contains mainly bug fixes to 1.39.24. Please more testing is still needed. ==== IMPORTANT new Win32 install procedure ===== - For Win32 migrations from versions prior to the new installer nothing special + For Win32 migrations from versions prior to 1.39.0 nothing special needs to be done to upgrade. Everything should be taken care of automatically. The only thing not done is to delete the old C:\Bacula directory (mostly out of paranoia, we can change that prior to release if everyone is comfortable with it). - Prior to using the current installer, if you used a prior 1.39.x - beta version, you must do one of the following three things: + Prior to using the current installer, if you used a version of the beta + prior to 1.39.26, you must do one of the following three things: 1) Uninstall prior beta version and select the option to delete the configuration and state files. This will get rid of the bogus - configuration files. Reinstall the new 1.39.22 version. + configuration files. Reinstall the new 1.39.26 version. 2) Manually copy the *.conf files from C:\Bacula\bin to C:\Documents and Settings\All Users\Application Data\Bacula. @@ -91,37 +91,6 @@ Version 1.39.26 contains mainly bug fixes to 1.39.24. Please After one of these has been done the upgrade should work fine. - If you have problems with the above, try the following - The new Win32 installer requires that you deinstall any - previous 1.39.x beta version before installing the new version. - The following is not necessary if you are upgrading from - a 1.38.x installation. - - The simplest way to do clean up a prior 1.39.x beta is in a DOS shell: - - cd c:\bacula\bin - net stop bacula - net stop baculfd - ./bacula-fd /remove - (Install the new Bacula) - - Note, one or both of the 'net stop' commands may fail. Don't - worry about it. - - If you didn't read the release notes or forgot to do those - commands, try the following: - - cd c:\bacula\bin - net stop bacula - net stop baculafd - sc delete bacula - sc delete baculafd - - Note, at least one of the 'net' commands and possibly both - will fail, and at least one of the 'sc' commands will fail. - Don't worry about it. After that, try again to install - the new Bacula. - New Features in 1.40.0: - Windows tray status windows are scrollable and resizable. @@ -179,7 +148,7 @@ New Features in 1.40.0: also compiles on Visual Studio, and at the same time, he added all the current Unix features to the FD, such as selection on drives, encryption support, building *all* the tools, ... - Finally, he also ported the Directory and the Storage daemon to + Finally, he also ported the Director and the Storage daemon to Win32. - Data encryption done in the Client is now supported due to code submitted by Landon Fuller. @@ -188,7 +157,7 @@ New Features in 1.40.0: the Microsoft specific permissions and ACLs will be lost. Thanks to Thorsten Engel for this code. - The 260 character limitation for Win32 paths name lengths is now - eliminated thanks to Thorsted Engel. + eliminated thanks to Thorsten Engel. - Eric Bollengier wrote new RunScript directives that includes the old RunBefore/AfterJob and ClientRunBefore/AfterJob features plus a *lot* more, allowing you to control just about every aspect @@ -281,8 +250,7 @@ New Features in 1.40.0: - Hash hard link filenames rather than linked list (performance). - Fix for security failure in chdir on Win32. - Add CreateDirectoryA/W win32 API entry points. -- Add /silent option to Win32 FD for Install/Remove service. - programs to duplicate Bacula's base64 algorithm using standard +- programs to duplicate Bacula's base64 algorithm using standard routines. This fixes bugs #296, and 565. Patch submitted by author of bug #565. - Fixes to reloading the Dir conf file from Eric Bollengier and Christopher @@ -373,18 +341,3 @@ WildDir = "[A-Z]:/{Documents and Settings,{WINNT,Windows}/Profiles}/*/{Local Settings,LOCALS~1}/Temp" WildDir = "[A-Z]:/{Documents and Settings,{WINNT,Windows}/Profiles}/*/{Local Settings,LOCALS~1}/Temporary Internet Files" - - - -Here are some hints if you want to play with the Windows Director and/or -Storage daemon. - -Changer and Tape device names in Windows are Changer0, Changer1, etc and -Tape0, Tape1, etc. If there isn't a device driver loaded for the Changer -then you need to use the address :::. Port is the -SCSI Adapter Number, Bus is the Bus Number on the adapter (usually 0 since -most adapters only have one bus), Target is the device's Target Device ID, -Lun is the Logical Unit Number. - -You must specify DeviceType = tape in the Device resource in bacula-sd.conf -since detection doesn't currently work. diff --git a/bacula/src/filed/backup.c b/bacula/src/filed/backup.c index f44fdda547..1fca1ec2ff 100644 --- a/bacula/src/filed/backup.c +++ b/bacula/src/filed/backup.c @@ -343,13 +343,14 @@ static int save_file(FF_PKT *ff_pkt, void *vjcr, bool top_level) // TODO landonf: We should really only calculate the digest once, for both verification and signing. if (jcr->pki_sign) { signing_digest = crypto_digest_new(signing_algorithm); - } - /* Full-stop if a failure occured initializing the signature digest */ - if (jcr->pki_sign && signing_digest == NULL) { - Jmsg(jcr, M_NOTSAVED, 0, _("%s signature digest initialization failed\n"), - stream_to_ascii(signing_algorithm)); - jcr->Errors++; - return 1; + + /* Full-stop if a failure occured initializing the signature digest */ + if (signing_digest == NULL) { + Jmsg(jcr, M_NOTSAVED, 0, _("%s signature digest initialization failed\n"), + stream_to_ascii(signing_algorithm)); + jcr->Errors++; + return 1; + } } /* Enable encryption */ @@ -599,7 +600,6 @@ int send_data(JCR *jcr, int stream, FF_PKT *ff_pkt, DIGEST *digest, DIGEST *sign wbuf = sd->msg; /* write buffer */ cipher_input = (uint8_t *)rbuf; /* encrypt uncompressed data */ - Dmsg1(300, "Saving data, type=%d\n", ff_pkt->type); #ifdef HAVE_LIBZ @@ -620,7 +620,7 @@ int send_data(JCR *jcr, int stream, FF_PKT *ff_pkt, DIGEST *digest, DIGEST *sign /* * Only change zlib parameters if there is no pending operation. - * This should never happen as deflaterset is called after each + * This should never happen as deflatereset is called after each * deflate. */ @@ -652,7 +652,7 @@ int send_data(JCR *jcr, int stream, FF_PKT *ff_pkt, DIGEST *digest, DIGEST *sign * could be returned for the given read buffer size. * (Using the larger of either rsize or max_compress_len) */ - jcr->crypto_buf = check_pool_memory_size(jcr->crypto_buf, (MAX(rsize, (int32_t)max_compress_len) + cipher_block_size - 1) / cipher_block_size * cipher_block_size); + jcr->crypto_buf = check_pool_memory_size(jcr->crypto_buf, (MAX(rsize + sizeof(uint32_t), (int32_t)max_compress_len) + cipher_block_size - 1) / cipher_block_size * cipher_block_size); wbuf = jcr->crypto_buf; /* Encrypted, possibly compressed output here. */ } @@ -670,7 +670,7 @@ int send_data(JCR *jcr, int stream, FF_PKT *ff_pkt, DIGEST *digest, DIGEST *sign /* * Make space at beginning of buffer for fileAddr because this - * same buffer will be used for writing if compression if off. + * same buffer will be used for writing if compression is off. */ if (ff_pkt->flags & FO_SPARSE) { rbuf += SPARSE_FADDR_SIZE; @@ -687,7 +687,7 @@ int send_data(JCR *jcr, int stream, FF_PKT *ff_pkt, DIGEST *digest, DIGEST *sign /* a RAW device read on win32 only works if the buffer is a multiple of 512 */ #ifdef HAVE_WIN32 if (S_ISBLK(ff_pkt->statp.st_mode)) - rsize = (rsize/512) * 512; + rsize = (rsize/512) * 512; #endif /* @@ -705,9 +705,10 @@ int send_data(JCR *jcr, int stream, FF_PKT *ff_pkt, DIGEST *digest, DIGEST *sign (uint64_t)ff_pkt->statp.st_size == 0)) { sparseBlock = is_buf_zero(rbuf, rsize); } - - ser_begin(wbuf, SPARSE_FADDR_SIZE); - ser_uint64(fileAddr); /* store fileAddr in begin of buffer */ + if (!sparseBlock) { + ser_begin(wbuf, SPARSE_FADDR_SIZE); + ser_uint64(fileAddr); /* store fileAddr in begin of buffer */ + } } jcr->ReadBytes += sd->msglen; /* count bytes read */ @@ -728,15 +729,14 @@ int send_data(JCR *jcr, int stream, FF_PKT *ff_pkt, DIGEST *digest, DIGEST *sign #ifdef HAVE_LIBZ /* Do compression if turned on */ - if (!sparseBlock && (ff_pkt->flags & FO_GZIP) && jcr->pZLIB_compress_workset) { - compress_len = max_compress_len; + if (!sparseBlock && (ff_pkt->flags & FO_GZIP) && jcr->pZLIB_compress_workset) { Dmsg4(400, "cbuf=0x%x len=%u rbuf=0x%x len=%u\n", cbuf, compress_len, rbuf, sd->msglen); ((z_stream*)jcr->pZLIB_compress_workset)->next_in = (Bytef *)rbuf; - ((z_stream*)jcr->pZLIB_compress_workset)->avail_in = sd->msglen; + ((z_stream*)jcr->pZLIB_compress_workset)->avail_in = sd->msglen; ((z_stream*)jcr->pZLIB_compress_workset)->next_out = (Bytef *)cbuf; - ((z_stream*)jcr->pZLIB_compress_workset)->avail_out = compress_len; + ((z_stream*)jcr->pZLIB_compress_workset)->avail_out = max_compress_len; if ((zstat=deflate((z_stream*)jcr->pZLIB_compress_workset, Z_FINISH)) != Z_STREAM_END) { Jmsg(jcr, M_FATAL, 0, _("Compression deflate error: %d\n"), zstat); @@ -751,24 +751,37 @@ int send_data(JCR *jcr, int stream, FF_PKT *ff_pkt, DIGEST *digest, DIGEST *sign goto err; } - Dmsg2(400, "compressed len=%d uncompressed len=%d\n", - compress_len, sd->msglen); + Dmsg2(400, "compressed len=%d uncompressed len=%d\n", compress_len, sd->msglen); sd->msglen = compress_len; /* set compressed length */ cipher_input_len = compress_len; } #endif - if (ff_pkt->flags & FO_ENCRYPT) { + if (!sparseBlock && (ff_pkt->flags & FO_ENCRYPT)) { + uint32_t initial_len = 0; + + if (ff_pkt->flags & FO_SPARSE) { + cipher_input_len += SPARSE_FADDR_SIZE; + } + + /* Encrypt the length of the input block */ + uint32_t packet_len = htonl(cipher_input_len); + + if (!crypto_cipher_update(cipher_ctx, (const u_int8_t *)&packet_len, sizeof(packet_len), (u_int8_t *)jcr->crypto_buf, &initial_len)) { + /* Encryption failed. Shouldn't happen. */ + Jmsg(jcr, M_FATAL, 0, _("Encryption error\n")); + goto err; + } + /* Encrypt the input block */ - if (crypto_cipher_update(cipher_ctx, cipher_input, cipher_input_len, (uint8_t *)jcr->crypto_buf, &encrypted_len)) { - if (encrypted_len == 0) { + if (crypto_cipher_update(cipher_ctx, cipher_input, cipher_input_len, (u_int8_t *)&jcr->crypto_buf[initial_len], &encrypted_len)) { + if ((initial_len + encrypted_len) == 0) { /* No full block of data available, read more data */ continue; } - Dmsg2(400, "encrypted len=%d unencrypted len=%d\n", - encrypted_len, sd->msglen); - sd->msglen = encrypted_len; /* set encrypted length */ + Dmsg2(400, "encrypted len=%d unencrypted len=%d\n", encrypted_len, sd->msglen); + sd->msglen = initial_len + encrypted_len; /* set encrypted length */ } else { /* Encryption failed. Shouldn't happen. */ Jmsg(jcr, M_FATAL, 0, _("Encryption error\n")); @@ -796,40 +809,35 @@ int send_data(JCR *jcr, int stream, FF_PKT *ff_pkt, DIGEST *digest, DIGEST *sign } /* end while read file data */ /* Send any remaining encrypted data + padding */ - if (ff_pkt->flags & FO_ENCRYPT) { - if (!crypto_cipher_finalize(cipher_ctx, (uint8_t *)jcr->crypto_buf, &encrypted_len)) { - /* Padding failed. Shouldn't happen. */ - Jmsg(jcr, M_FATAL, 0, _("Encryption padding error\n")); - goto err; - } + if (sd->msglen >= 0) { + if (ff_pkt->flags & FO_ENCRYPT) { + if (!crypto_cipher_finalize(cipher_ctx, (uint8_t *)jcr->crypto_buf, &encrypted_len)) { + /* Padding failed. Shouldn't happen. */ + Jmsg(jcr, M_FATAL, 0, _("Encryption padding error\n")); + goto err; + } - if (encrypted_len > 0) { - sd->msglen = encrypted_len; /* set encrypted length */ + if (encrypted_len > 0) { + sd->msglen = encrypted_len; /* set encrypted length */ - /* Send remaining encrypted data to the SD */ - if (ff_pkt->flags & FO_SPARSE) { - sd->msglen += SPARSE_FADDR_SIZE; /* include fileAddr in size */ - } - sd->msg = wbuf; /* set correct write buffer */ - if (!bnet_send(sd)) { - Jmsg1(jcr, M_FATAL, 0, _("Network send error to SD. ERR=%s\n"), - bnet_strerror(sd)); - goto err; + sd->msg = jcr->crypto_buf; /* set correct write buffer */ + if (!bnet_send(sd)) { + Jmsg1(jcr, M_FATAL, 0, _("Network send error to SD. ERR=%s\n"), + bnet_strerror(sd)); + goto err; + } + Dmsg1(130, "Send data to SD len=%d\n", sd->msglen); + jcr->JobBytes += sd->msglen; /* count bytes saved possibly compressed/encrypted */ + sd->msg = msgsave; /* restore bnet buffer */ } - Dmsg1(130, "Send data to SD len=%d\n", sd->msglen); - jcr->JobBytes += sd->msglen; /* count bytes saved possibly compressed/encrypted */ - sd->msg = msgsave; /* restore bnet buffer */ } - } - - if (sd->msglen < 0) { + } else { berrno be; Jmsg(jcr, M_ERROR, 0, _("Read error on file %s. ERR=%s\n"), ff_pkt->fname, be.strerror(ff_pkt->bfd.berrno)); if (jcr->Errors++ > 1000) { /* insanity check */ Jmsg(jcr, M_FATAL, 0, _("Too many errors.\n")); } - } if (!bnet_sig(sd, BNET_EOD)) { /* indicate end of file data */ diff --git a/bacula/src/filed/restore.c b/bacula/src/filed/restore.c index c10e1225da..5f9238609f 100644 --- a/bacula/src/filed/restore.c +++ b/bacula/src/filed/restore.c @@ -350,11 +350,11 @@ void do_restore(JCR *jcr) Dmsg1(30, "Stream=Encrypted Session Data, size: %d\n", sd->msglen); - /* Do we have any keys at all? */ - if (!jcr->pki_recipients) { - Jmsg(jcr, M_ERROR, 0, _("No private decryption keys have been defined to decrypt encrypted backup data.")); - break; - } + /* Do we have any keys at all? */ + if (!jcr->pki_recipients) { + Jmsg(jcr, M_ERROR, 0, _("No private decryption keys have been defined to decrypt encrypted backup data.")); + break; + } /* Decode and save session keys. */ cryptoerr = crypto_session_decode((uint8_t *)sd->msg, (uint32_t)sd->msglen, jcr->pki_recipients, &cs); @@ -389,6 +389,9 @@ void do_restore(JCR *jcr) bclose(&bfd); continue; } + + jcr->crypto_count = 0; + jcr->crypto_size = 0; break; case STREAM_FILE_DATA: @@ -734,6 +737,55 @@ int verify_signature(JCR *jcr, SIGNATURE *sig) return false; } +bool decompress_data(JCR *jcr, char **data, uint32_t *length) +{ +#ifdef HAVE_LIBZ + uLong compress_len; + int stat; + char ec1[50]; /* Buffer printing huge values */ + + /* + * NOTE! We only use uLong and Byte because they are + * needed by the zlib routines, they should not otherwise + * be used in Bacula. + */ + compress_len = jcr->compress_buf_size; + Dmsg2(100, "Comp_len=%d msglen=%d\n", compress_len, *length); + if ((stat=uncompress((Byte *)jcr->compress_buf, &compress_len, + (const Byte *)*data, (uLong)*length)) != Z_OK) { + Qmsg(jcr, M_ERROR, 0, _("Uncompression error on file %s. ERR=%s\n"), + jcr->last_fname, zlib_strerror(stat)); + return false; + } + *data = jcr->compress_buf; + *length = compress_len; + Dmsg2(100, "Write uncompressed %d bytes, total before write=%s\n", compress_len, edit_uint64(jcr->JobBytes, ec1)); + return true; +#else + Qmsg(jcr, M_ERROR, 0, _("GZIP data stream found, but GZIP not configured!\n")); + return false; +#endif +} + +bool store_data(JCR *jcr, BFILE *bfd, char *data, const int32_t length, bool win32_decomp) +{ + if (win32_decomp) { + if (!processWin32BackupAPIBlock(bfd, data, length)) { + berrno be; + Jmsg2(jcr, M_ERROR, 0, _("Write error in Win32 Block Decomposition on %s: %s\n"), + jcr->last_fname, be.strerror(bfd->berrno)); + return false; + } + } else if (bwrite(bfd, data, length) != (ssize_t)length) { + berrno be; + Jmsg2(jcr, M_ERROR, 0, _("Write error on %s: %s\n"), + jcr->last_fname, be.strerror(bfd->berrno)); + return false; + } + + return true; +} + /* * In the context of jcr, write data to bfd. * We write buflen bytes in buf at addr. addr is updated in place. @@ -743,123 +795,147 @@ int verify_signature(JCR *jcr, SIGNATURE *sig) int32_t extract_data(JCR *jcr, BFILE *bfd, POOLMEM *buf, int32_t buflen, uint64_t *addr, int flags, CIPHER_CONTEXT *cipher, uint32_t cipher_block_size) { - int stat; char *wbuf; /* write buffer */ uint32_t wsize; /* write size */ uint32_t rsize; /* read size */ - char ec1[50]; /* Buffer printing huge values */ - const uint8_t *cipher_input; /* Decryption input */ - uint32_t cipher_input_len; /* Decryption input length */ uint32_t decrypted_len = 0; /* Decryption output length */ + char ec1[50]; /* Buffer printing huge values */ - if (flags & FO_SPARSE) { - ser_declare; - uint64_t faddr; - char ec1[50]; - wbuf = buf + SPARSE_FADDR_SIZE; - rsize = buflen - SPARSE_FADDR_SIZE; - ser_begin(buf, SPARSE_FADDR_SIZE); - unser_uint64(faddr); - if (*addr != faddr) { - *addr = faddr; - if (blseek(bfd, (off_t)*addr, SEEK_SET) < 0) { - berrno be; - Jmsg3(jcr, M_ERROR, 0, _("Seek to %s error on %s: ERR=%s\n"), - edit_uint64(*addr, ec1), jcr->last_fname, - be.strerror(bfd->berrno)); - return -1; - } - } - } else { - wbuf = buf; - rsize = buflen; - } + rsize = buflen; + jcr->ReadBytes += rsize; wsize = rsize; - cipher_input = (uint8_t *)wbuf; - cipher_input_len = (uint32_t)wsize; - - if (flags & FO_GZIP) { -#ifdef HAVE_LIBZ - uLong compress_len; - /* - * NOTE! We only use uLong and Byte because they are - * needed by the zlib routines, they should not otherwise - * be used in Bacula. - */ - compress_len = jcr->compress_buf_size; - Dmsg2(100, "Comp_len=%d msglen=%d\n", compress_len, wsize); - if ((stat=uncompress((Byte *)jcr->compress_buf, &compress_len, - (const Byte *)wbuf, (uLong)rsize)) != Z_OK) { - Qmsg(jcr, M_ERROR, 0, _("Uncompression error on file %s. ERR=%s\n"), - jcr->last_fname, zlib_strerror(stat)); - return -1; - } - wbuf = jcr->compress_buf; - wsize = compress_len; - cipher_input = (uint8_t *)jcr->compress_buf; /* decrypt decompressed data */ - cipher_input_len = compress_len; - Dmsg2(100, "Write uncompressed %d bytes, total before write=%s\n", compress_len, edit_uint64(jcr->JobBytes, ec1)); -#else - Qmsg(jcr, M_ERROR, 0, _("GZIP data stream found, but GZIP not configured!\n")); - return -1; -#endif - } else { - Dmsg2(30, "Write %u bytes, total before write=%s\n", wsize, edit_uint64(jcr->JobBytes, ec1)); - } + wbuf = buf; if (flags & FO_ENCRYPT) { ASSERT(cipher); + while (jcr->crypto_size > 0 && jcr->crypto_count > 0 && wsize > 0) { + uint32_t chunk_size = 16; + + if (chunk_size > wsize) { + chunk_size = wsize; + } + + /* + * Grow the crypto buffer, if necessary. + * crypto_cipher_update() will process only whole blocks, + * buffering the remaining input. + */ + jcr->crypto_buf = check_pool_memory_size(jcr->crypto_buf, jcr->crypto_count + chunk_size + cipher_block_size); + + /* Decrypt the input block */ + if (!crypto_cipher_update(cipher, + (const u_int8_t *)wbuf, + chunk_size, + (u_int8_t *)&jcr->crypto_buf[jcr->crypto_count], + &decrypted_len)) { + /* Decryption failed. Shouldn't happen. */ + Jmsg(jcr, M_FATAL, 0, _("Decryption error\n")); + return -1; + } + + jcr->crypto_count += decrypted_len; + wbuf += chunk_size; + wsize -= chunk_size; + + if (jcr->crypto_count >= jcr->crypto_size) { + + char *packet = &jcr->crypto_buf[4]; /* Decrypted, possibly decompressed output here. */ + uint32_t packet_size = jcr->crypto_size - 4; + + if (flags & FO_GZIP) { + if (!decompress_data(jcr, &packet, &packet_size)) { + return -1; + } + } else { + Dmsg2(30, "Write %u bytes, total before write=%s\n", wsize, edit_uint64(jcr->JobBytes, ec1)); + } + + if (!store_data(jcr, bfd, packet, packet_size, (flags & FO_WIN32DECOMP) != 0)) { + return -1; + } + + jcr->JobBytes += packet_size; + *addr += packet_size; + + memmove(&jcr->crypto_buf[0], &jcr->crypto_buf[jcr->crypto_size], jcr->crypto_count - jcr->crypto_size); + jcr->crypto_count -= jcr->crypto_size; + jcr->crypto_size = 0; + } + } + /* * Grow the crypto buffer, if necessary. * crypto_cipher_update() will process only whole blocks, * buffering the remaining input. */ - jcr->crypto_buf = check_pool_memory_size(jcr->crypto_buf, cipher_input_len + cipher_block_size); - - - /* Encrypt the input block */ - if (!crypto_cipher_update(cipher, cipher_input, cipher_input_len, (uint8_t *)jcr->crypto_buf, &decrypted_len)) { + jcr->crypto_buf = check_pool_memory_size(jcr->crypto_buf, jcr->crypto_count + wsize + cipher_block_size); + + /* Decrypt the input block */ + if (!crypto_cipher_update(cipher, + (const u_int8_t *)wbuf, + wsize, + (u_int8_t *)&jcr->crypto_buf[jcr->crypto_count], + &decrypted_len)) { /* Decryption failed. Shouldn't happen. */ Jmsg(jcr, M_FATAL, 0, _("Decryption error\n")); return -1; } + Dmsg2(100, "decrypted len=%d encrypted len=%d\n", decrypted_len, wsize); + if (decrypted_len == 0) { /* No full block of data available, write more data */ - goto ok; + return 0; + } + + jcr->crypto_count += decrypted_len; + + if (jcr->crypto_size == 0 && jcr->crypto_count >= 4) { + jcr->crypto_size = ntohl(*(uint32_t *)&jcr->crypto_buf[0]) + 4; + } + + if (jcr->crypto_size == 0 || jcr->crypto_count < jcr->crypto_size) { + return 0; } - Dmsg2(400, "decrypted len=%d undecrypted len=%d\n", - decrypted_len, cipher_input_len); - wsize = decrypted_len; - wbuf = jcr->crypto_buf; /* Decrypted, possibly decompressed output here. */ + wsize = jcr->crypto_size - 4; + wbuf = &jcr->crypto_buf[4]; /* Decrypted, possibly decompressed output here. */ } + if (flags & FO_SPARSE) { + ser_declare; + uint64_t faddr; + char ec1[50]; + ser_begin(wbuf, SPARSE_FADDR_SIZE); + unser_uint64(faddr); + if (*addr != faddr) { + *addr = faddr; + if (blseek(bfd, (off_t)*addr, SEEK_SET) < 0) { + berrno be; + Jmsg3(jcr, M_ERROR, 0, _("Seek to %s error on %s: ERR=%s\n"), + edit_uint64(*addr, ec1), jcr->last_fname, + be.strerror(bfd->berrno)); + return -1; + } + } + wbuf += SPARSE_FADDR_SIZE; + wsize -= SPARSE_FADDR_SIZE; + } - if (flags & FO_WIN32DECOMP) { - if (!processWin32BackupAPIBlock(bfd, wbuf, wsize)) { - berrno be; - Jmsg2(jcr, M_ERROR, 0, _("Write error in Win32 Block Decomposition on %s: %s\n"), - jcr->last_fname, be.strerror(bfd->berrno)); + if (flags & FO_GZIP) { + if (!decompress_data(jcr, &wbuf, &wsize)) { return -1; } - } else if (bwrite(bfd, wbuf, wsize) != (ssize_t)wsize) { - berrno be; - Jmsg2(jcr, M_ERROR, 0, _("Write error on %s: %s\n"), - jcr->last_fname, be.strerror(bfd->berrno)); - return -1; + } else { + Dmsg2(30, "Write %u bytes, total before write=%s\n", wsize, edit_uint64(jcr->JobBytes, ec1)); } - if (decrypted_len && decrypted_len > wsize) { - /* If more than wsize is output, it was previously buffered - * and reported, and should not be reported again */ - wsize = wsize - decrypted_len; + if (!store_data(jcr, bfd, wbuf, wsize, (flags & FO_WIN32DECOMP) != 0)) { + return -1; } -ok: jcr->JobBytes += wsize; - jcr->ReadBytes += rsize; *addr += wsize; return wsize; @@ -873,28 +949,47 @@ ok: bool flush_cipher(JCR *jcr, BFILE *bfd, int flags, CIPHER_CONTEXT *cipher, uint32_t cipher_block_size) { uint32_t decrypted_len; + char *wbuf; /* write buffer */ + uint32_t wsize; /* write size */ + char ec1[50]; /* Buffer printing huge values */ /* Write out the remaining block and free the cipher context */ - jcr->crypto_buf = check_pool_memory_size(jcr->crypto_buf, cipher_block_size); + jcr->crypto_buf = check_pool_memory_size(jcr->crypto_buf, jcr->crypto_count + cipher_block_size); - if (!crypto_cipher_finalize(cipher, (uint8_t *)jcr->crypto_buf, &decrypted_len)) { + if (!crypto_cipher_finalize(cipher, (uint8_t *)&jcr->crypto_buf[jcr->crypto_count], &decrypted_len)) { /* Writing out the final, buffered block failed. Shouldn't happen. */ Jmsg1(jcr, M_FATAL, 0, _("Decryption error for %s\n"), jcr->last_fname); } - if (flags & FO_WIN32DECOMP) { - if (!processWin32BackupAPIBlock(bfd, jcr->crypto_buf, decrypted_len)) { - berrno be; - Jmsg2(jcr, M_ERROR, 0, _("Write error in Win32 Block Decomposition on %s: %s\n"), - jcr->last_fname, be.strerror(bfd->berrno)); - return false; - } - } else if (bwrite(bfd, jcr->crypto_buf, decrypted_len) != (ssize_t)decrypted_len) { - berrno be; - Jmsg2(jcr, M_ERROR, 0, _("Write error on %s: %s\n"), - jcr->last_fname, be.strerror(bfd->berrno)); + if (decrypted_len == 0) + { + ASSERT(jcr->crypto_count == 0); + return true; + } + + jcr->crypto_count += decrypted_len; + + if (jcr->crypto_size == 0) { + ASSERT(jcr->crypto_count >= 4); + jcr->crypto_size = ntohl(*(uint32_t *)&jcr->crypto_buf[0]) + 4; + } + + ASSERT(jcr->crypto_count == jcr->crypto_size); + + wbuf = &jcr->crypto_buf[4]; + wsize = jcr->crypto_size - 4; + + if (flags & FO_GZIP) { + decompress_data(jcr, &wbuf, &wsize); + } else { + Dmsg2(30, "Write %u bytes, total before write=%s\n", wsize, edit_uint64(jcr->JobBytes, ec1)); + } + + if (!store_data(jcr, bfd, wbuf, wsize, (flags & FO_WIN32DECOMP) != 0)) { return false; } + jcr->JobBytes += wsize; + return true; } diff --git a/bacula/src/jcr.h b/bacula/src/jcr.h index c13da9f9da..f71527e172 100644 --- a/bacula/src/jcr.h +++ b/bacula/src/jcr.h @@ -275,6 +275,8 @@ public: uint8_t *pki_session_encoded; /* Cached DER-encoded copy of pki_session */ int32_t pki_session_encoded_size; /* Size of DER-encoded pki_session */ POOLMEM *crypto_buf; /* Encryption/Decryption buffer */ + int32_t crypto_count; /* Count of bytes currently in crypto_buf */ + int32_t crypto_size; /* Total bytes in packet */ DIRRES* director; /* Director resource */ bool runscript_after; /* Don't run After Script twice */ #endif /* FILE_DAEMON */ diff --git a/bacula/src/lib/berrno.h b/bacula/src/lib/berrno.h index 46a6a3a304..efc6af05c8 100644 --- a/bacula/src/lib/berrno.h +++ b/bacula/src/lib/berrno.h @@ -37,7 +37,7 @@ * it is thread safe. * * If bit 29 in berrno_ is set then it is a Win32 error, and we - * must to a GetLastError() to get the error code for formatting. + * must do a GetLastError() to get the error code for formatting. * If bit 29 in berrno_ is not set, then it is a Unix errno. * */ diff --git a/bacula/src/lib/bsys.c b/bacula/src/lib/bsys.c index 2869a3fa29..9dd7c96c3a 100644 --- a/bacula/src/lib/bsys.c +++ b/bacula/src/lib/bsys.c @@ -705,8 +705,8 @@ char *bfgets(char *s, int size, FILE *fd) do { errno = 0; ch = fgetc(fd); - } while (ch == -1 && (errno == EINTR || errno == EAGAIN)); - if (ch == -1) { + } while (ch == EOF && ferror(fd) && (errno == EINTR || errno == EAGAIN)); + if (ch == EOF) { if (i == 0) { return NULL; } else { @@ -717,13 +717,10 @@ char *bfgets(char *s, int size, FILE *fd) *p = 0; if (ch == '\r') { /* Support for Mac/Windows file format */ ch = fgetc(fd); - if (ch == '\n') { /* Windows (\r\n) */ - *p++ = ch; - *p = 0; - } - else { /* Mac (\r only) */ + if (ch != '\n') { /* Mac (\r only) */ (void)ungetc(ch, fd); /* Push next character back to fd */ } + p[-1] = '\n'; break; } if (ch == '\n') { diff --git a/bacula/src/tools/bsmtp.c b/bacula/src/tools/bsmtp.c index 75cbdcdbfb..9cccc49c29 100644 --- a/bacula/src/tools/bsmtp.c +++ b/bacula/src/tools/bsmtp.c @@ -216,6 +216,10 @@ int main (int argc, char *argv[]) exit(1); } +#if defined(HAVE_WIN32) + _setmode(0, _O_BINARY); +#endif + /* * Determine SMTP server */ @@ -314,7 +318,7 @@ hp: Dmsg0(20, "Connected\n"); #if defined(HAVE_WIN32) - int fdSocket = _open_osfhandle(s, _O_RDWR); + int fdSocket = _open_osfhandle(s, _O_RDWR | _O_BINARY); if (fdSocket == -1) { Pmsg1(0, _("Fatal _open_osfhandle error: ERR=%s\n"), strerror(errno)); exit(1); @@ -449,11 +453,13 @@ __MINGW_IMPORT long _dstbias; Dmsg1(20, "skip line because of maxlines limit: %lu\n", maxlines); break; } - buf[strlen(buf)-1] = 0; - if (strcmp(buf, ".") == 0) { /* quote lone dots */ - fprintf(sfp, "..\r\n"); + buf[sizeof(buf)-1] = '\0'; + buf[strlen(buf)-1] = '\0'; + if (buf[0] == '.' && buf[1] == '\0') { /* quote lone dots */ + fputs("..\r\n", sfp); } else { /* pass body through unchanged */ - fprintf(sfp, "%s\r\n", buf); + fputs(buf, sfp); + fputs("\r\n", sfp); } } diff --git a/bacula/src/win32/bacula.sln b/bacula/src/win32/bacula.sln index b8684d9b3c..cc768f2ed9 100644 --- a/bacula/src/win32/bacula.sln +++ b/bacula/src/win32/bacula.sln @@ -52,6 +52,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution README.mingw32 = README.mingw32 README.vc8 = README.vc8 README.win32 = README.win32 + ..\..\technotes-1.39 = ..\..\technotes-1.39 EndProjectSection EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Header Files", "Header Files", "{C8301485-CFD1-43D4-827C-8EA050C8E256}" @@ -160,6 +161,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "cats_bdb", "cats\cats_bdb\c EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "installer", "installer\installer.vcproj", "{6D1B0964-FB32-4916-A61C-49D7F715EAD8}" ProjectSection(ProjectDependencies) = postProject + {A0F65E06-9F18-40AC-81F6-A080852F1104} = {A0F65E06-9F18-40AC-81F6-A080852F1104} {9BA8E10D-0D82-4B25-8543-DE34641FBC10} = {9BA8E10D-0D82-4B25-8543-DE34641FBC10} {614CE916-0972-4126-9392-CD9FC0ADD7DE} = {614CE916-0972-4126-9392-CD9FC0ADD7DE} {85696E20-777A-41F6-BC00-2E7AB375B171} = {85696E20-777A-41F6-BC00-2E7AB375B171} @@ -170,6 +172,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "installer", "installer\inst {F8AF7D74-2918-422B-A7B6-4D98566B7160} = {F8AF7D74-2918-422B-A7B6-4D98566B7160} {374BF775-AF68-4A88-814A-48F692DFFE5A} = {374BF775-AF68-4A88-814A-48F692DFFE5A} {6A7AA493-E46C-4994-B8D6-AA6C9C19C9BA} = {6A7AA493-E46C-4994-B8D6-AA6C9C19C9BA} + {AB67F297-8491-4515-8E52-BFF5340EC242} = {AB67F297-8491-4515-8E52-BFF5340EC242} {2D729599-C008-4154-BCCB-53E6A260F220} = {2D729599-C008-4154-BCCB-53E6A260F220} {0F56AEB0-14DA-4A80-8962-1F85A17339D0} = {0F56AEB0-14DA-4A80-8962-1F85A17339D0} {8B79A2B5-8889-43D4-9B92-9AE8A6F00413} = {8B79A2B5-8889-43D4-9B92-9AE8A6F00413} @@ -183,8 +186,6 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "installer", "installer\inst {D03415F7-654E-42F4-B0E9-CB8FBE3F22FA} = {D03415F7-654E-42F4-B0E9-CB8FBE3F22FA} {F5F063F8-11A1-475A-82E2-19759BB40B25} = {F5F063F8-11A1-475A-82E2-19759BB40B25} {558838F9-D792-4F56-AAB2-99C03687C5FF} = {558838F9-D792-4F56-AAB2-99C03687C5FF} - {AB67F297-8491-4515-8E52-BFF5340EC242} = {AB67F297-8491-4515-8E52-BFF5340EC242} - {A0F65E06-9F18-40AC-81F6-A080852F1104} = {A0F65E06-9F18-40AC-81F6-A080852F1104} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "bsleep", "scripts\bsleep.vcproj", "{0F56AEB0-14DA-4A80-8962-1F85A17339D0}" diff --git a/bacula/src/win32/cats/cats_mysql/cats_mysql.vcproj b/bacula/src/win32/cats/cats_mysql/cats_mysql.vcproj index 150d15b467..9b1372dfcc 100644 --- a/bacula/src/win32/cats/cats_mysql/cats_mysql.vcproj +++ b/bacula/src/win32/cats/cats_mysql/cats_mysql.vcproj @@ -382,10 +382,6 @@ UniqueIdentifier="{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}" > - - diff --git a/bacula/src/win32/cats/create_mysql_database.cmd b/bacula/src/win32/cats/create_mysql_database.cmd index f6bb8c8ffd..bc079b1d7f 100644 --- a/bacula/src/win32/cats/create_mysql_database.cmd +++ b/bacula/src/win32/cats/create_mysql_database.cmd @@ -1,8 +1,9 @@ +@echo off rem rem Script to create Bacula database(s) rem -%SQL_BINDIR%\mysql $* -e "CREATE DATABASE bacula;" +"%SQL_BINDIR%\mysql" %* -e "CREATE DATABASE bacula;" set RESULT=%ERRORLEVEL% if %RESULT% GTR 0 goto :ERROR echo "Creation of bacula database succeeded." diff --git a/bacula/src/win32/cats/drop_mysql_database.cmd b/bacula/src/win32/cats/drop_mysql_database.cmd index ddee47cec7..7289c53773 100644 --- a/bacula/src/win32/cats/drop_mysql_database.cmd +++ b/bacula/src/win32/cats/drop_mysql_database.cmd @@ -2,7 +2,7 @@ rem rem shell script to drop Bacula database(s) rem -%SQL_BINDIR%/mysql $* -f -e "DROP DATABASE bacula;" +"%SQL_BINDIR%/mysql" %* -f -e "DROP DATABASE bacula;" set RESULT=%ERRORLEVEL% if %RESULT% GTR 0 goto :ERROR echo "Drop of bacula database succeeded." diff --git a/bacula/src/win32/cats/drop_mysql_tables.cmd b/bacula/src/win32/cats/drop_mysql_tables.cmd index 7e2640fe65..0dd328ad55 100644 --- a/bacula/src/win32/cats/drop_mysql_tables.cmd +++ b/bacula/src/win32/cats/drop_mysql_tables.cmd @@ -1,8 +1,9 @@ +@echo off rem rem Script to delete Bacula tables for MySQL rem -if %SQL_BINDIR%/mysql $* < drop_mysql_tables.sql +"%SQL_BINDIR%/mysql" %* < drop_mysql_tables.sql set RESULT=%ERRORLEVEL% if %RESULT% GTR 0 goto :ERROR echo "Deletion of Bacula MySQL tables succeeded." diff --git a/bacula/src/win32/cats/grant_mysql_privileges.cmd b/bacula/src/win32/cats/grant_mysql_privileges.cmd index 85204ad441..e94f3af3bc 100644 --- a/bacula/src/win32/cats/grant_mysql_privileges.cmd +++ b/bacula/src/win32/cats/grant_mysql_privileges.cmd @@ -1,8 +1,9 @@ +@echo off rem rem Script to grant privileges to the bacula database rem -%SQL_BINDIR%\mysql $* -u root -f < grant_mysql_privileges.sql +"%SQL_BINDIR%\mysql" -u root -f %* < grant_mysql_privileges.sql set RESULT=%ERRORLEVEL% if %RESULT% GTR 0 goto :ERROR echo "Privileges for bacula granted." diff --git a/bacula/src/win32/cats/make_mysql_tables.cmd b/bacula/src/win32/cats/make_mysql_tables.cmd index 5e73de3352..1d3906f7a0 100644 --- a/bacula/src/win32/cats/make_mysql_tables.cmd +++ b/bacula/src/win32/cats/make_mysql_tables.cmd @@ -1,10 +1,11 @@ +@echo off rem rem Script to create Bacula MySQL tables rem -%SQL_BINDIR%\mysql -f < make_mysql_tables.sql +"%SQL_BINDIR%\mysql" -f %* < make_mysql_tables.sql set RESULT=%ERRORLEVEL% -if %RESULT% gt 0 goto :ERROR +if %RESULT% GTR 0 goto :ERROR echo "Creation of Bacula MySQL tables succeeded." exit /b 0 diff --git a/bacula/src/win32/cats/update_mysql_tables.cmd b/bacula/src/win32/cats/update_mysql_tables.cmd index 3baf987091..377d4a1c55 100644 --- a/bacula/src/win32/cats/update_mysql_tables.cmd +++ b/bacula/src/win32/cats/update_mysql_tables.cmd @@ -8,7 +8,7 @@ echo Depending on the size of your database, echo this script may take several minutes to run. echo. -"%SQL_BINDIR%\mysql" %* -f -u bacula --password=bacula bacula < update_mysql_tables.sql +"%SQL_BINDIR%\mysql" %* -f -u bacula bacula < update_mysql_tables.sql set RESULT=%ERRORLEVEL% if %RESULT% GTR 0 goto :ERROR echo "Update of Bacula MySQL tables succeeded." diff --git a/bacula/src/win32/compat/compat.cpp b/bacula/src/win32/compat/compat.cpp index 001e82bc57..332cb64ca8 100644 --- a/bacula/src/win32/compat/compat.cpp +++ b/bacula/src/win32/compat/compat.cpp @@ -1915,18 +1915,18 @@ open_bpipe(char *prog, int wait, const char *mode) // process terminates we can // detect eof. // ugly but convert WIN32 HANDLE to FILE* - int rfd = _open_osfhandle((long)hChildStdoutRdDup, O_RDONLY); + int rfd = _open_osfhandle((long)hChildStdoutRdDup, O_RDONLY | O_BINARY); if (rfd >= 0) { - bpipe->rfd = _fdopen(rfd, "r"); + bpipe->rfd = _fdopen(rfd, "rb"); } } if (mode_write) { CloseHandle(hChildStdinRd); // close our read side so as not // to interfre with child's copy // ugly but convert WIN32 HANDLE to FILE* - int wfd = _open_osfhandle((long)hChildStdinWrDup, O_WRONLY); + int wfd = _open_osfhandle((long)hChildStdinWrDup, O_WRONLY | O_BINARY); if (wfd >= 0) { - bpipe->wfd = _fdopen(wfd, "w"); + bpipe->wfd = _fdopen(wfd, "wb"); } } @@ -1968,6 +1968,16 @@ close_bpipe(BPIPE *bpipe) int rval = 0; int32_t remaining_wait = bpipe->wait; + /* Close pipes */ + if (bpipe->rfd) { + fclose(bpipe->rfd); + bpipe->rfd = NULL; + } + if (bpipe->wfd) { + fclose(bpipe->wfd); + bpipe->wfd = NULL; + } + if (remaining_wait == 0) { /* wait indefinitely */ remaining_wait = INT32_MAX; } @@ -2008,16 +2018,16 @@ close_bpipe(BPIPE *bpipe) int close_wpipe(BPIPE *bpipe) { - int stat = 1; + int result = 1; if (bpipe->wfd) { fflush(bpipe->wfd); if (fclose(bpipe->wfd) != 0) { - stat = 0; + result = 0; } bpipe->wfd = NULL; } - return stat; + return result; } #include "findlib/find.h" diff --git a/bacula/technotes-1.39 b/bacula/technotes-1.39 index 2db8d0ff4a..4cb71989f1 100644 --- a/bacula/technotes-1.39 +++ b/bacula/technotes-1.39 @@ -2,6 +2,15 @@ General: +05Nov06 +rbn Fixed problems with encryption when combined with compression + or sparse files. Unfortunately this means that all previous + encrypted backups can't be restored. +rbn Fixed problem in bfgets with Windows and Mac end of lines. +rbn Fixed bug in Windows version of close_bpipe() that cause programs + like bsmtp() to hang waiting for the pipe to be closed. Fixed + binary file type problems in open_bpipe(). +rbn Fixed a number of bugs in the mysql DB scripts. 03Nov06 ebl Fix broken runscript on director. (RunBefore, RunAfter) ebl Use M_FATAL instead of M_ERROR in runscript.c