I took my kayak out to Rainbow Springs State Park today to get in some paddling. After a long, hard winter (I think we were below freezing three times), I wanted to get comfortable in the boat before heading up to GA/TN/NC next weekend for three days of paddling. Even though I can't take vacation yet, the powers that be decided to give me a day off since work's been crazy lately. I'll be meeting up with the Spring Break trip from Madison.
Whoo boating!
Sunday, March 25, 2007
Tuesday, March 6, 2007
Samba upgrade part 2
We sleep, we wake, we go back to work.
When we get into work this morning we find out that some things had fallen out of the domain in all the fscking around last night. And we still don't have a working machine add to put them back in. Curse, experiment, discuss... The decision is made to try a home build of the newer Samba 3, instead of the downloaded binaries, and we are assured that all the paths are the same and everything should just work. Install, restart, and things go completely sideways. Stock samba is reinstalled, LDAP restored from a slave, and everything restarted.
This time, many things have fallen out of the domain, including our two main samba file servers. Eventually, a technique is discovered to restore some of the Windows domain trusts. It turns out that the LDAP slave chosen for the restore didn't have all the machine password changes. Restoring passwords from a different slave allowed some machines to rejoin. This worked for one of the file servers, but not the other. I spent most of the afternoon trying to get the other server back into the domain. I found an unadvertised tool to take apart the samba secrets.tdb file, but the failed server is running 2.2.8 and the working server is running 3.<mumble>, so comparing the two is only marginally helpful. Late in the afternoon, we are able to manually beat the other server back into the domain.
Things start to look vaguely better and all we're left with is these Windows machines which have fallen out of the domain. People start heading home, and only myself and our Team Lead are left sweeping up.
We're discussing the state of affairs, and I mention in stream of conciousness mode, "So, we need a samba account that maps to UNIX uid 0. The question is does <samba domain admin> have a UNIX..."
Team Lead: "Aaagh!"
Me: "...account?" The team lead turns to his computer and begins typing furiously. I start thinking while he types and realize my question is nonsensical. <samba domain admin> is in LDAP as a POSIX (and samba) account with uid 0, so UNIX will see it as uid 0. Team Lead trys a domain join and Success!
It turns out that while the sambaLMPassword and sambaNTPassword were set, the UNIX password in LDAP isn't. Apparently, this is why the account kept being disabled and the passwords deleted.
Gah! At least things are working now.
When we get into work this morning we find out that some things had fallen out of the domain in all the fscking around last night. And we still don't have a working machine add to put them back in. Curse, experiment, discuss... The decision is made to try a home build of the newer Samba 3, instead of the downloaded binaries, and we are assured that all the paths are the same and everything should just work. Install, restart, and things go completely sideways. Stock samba is reinstalled, LDAP restored from a slave, and everything restarted.
This time, many things have fallen out of the domain, including our two main samba file servers. Eventually, a technique is discovered to restore some of the Windows domain trusts. It turns out that the LDAP slave chosen for the restore didn't have all the machine password changes. Restoring passwords from a different slave allowed some machines to rejoin. This worked for one of the file servers, but not the other. I spent most of the afternoon trying to get the other server back into the domain. I found an unadvertised tool to take apart the samba secrets.tdb file, but the failed server is running 2.2.8 and the working server is running 3.<mumble>, so comparing the two is only marginally helpful. Late in the afternoon, we are able to manually beat the other server back into the domain.
Things start to look vaguely better and all we're left with is these Windows machines which have fallen out of the domain. People start heading home, and only myself and our Team Lead are left sweeping up.
We're discussing the state of affairs, and I mention in stream of conciousness mode, "So, we need a samba account that maps to UNIX uid 0. The question is does <samba domain admin> have a UNIX..."
Team Lead: "Aaagh!"
Me: "...account?" The team lead turns to his computer and begins typing furiously. I start thinking while he types and realize my question is nonsensical. <samba domain admin> is in LDAP as a POSIX (and samba) account with uid 0, so UNIX will see it as uid 0. Team Lead trys a domain join and Success!
It turns out that while the sambaLMPassword and sambaNTPassword were set, the UNIX password in LDAP isn't. Apparently, this is why the account kept being disabled and the passwords deleted.
Gah! At least things are working now.
Samba upgrade
One of the medium term projects at work is to move towards Active Directory. Why, you might ask? Supposedly AD would solve some problems that non-Gainesville employees are having, such as password changing. We have a password change after 180(?) days policy and employees on site in many states, so this needs to work.
Anyway, part one is to update our Samba servers to version 3 because AD won't form a domain trust with earlier versions. Last night was the night we updated the Primary Domain Controller. Since I don't know much about Samba, I was mostly along for the ride.
Of course, it's never that simple. Under Samba 2, we were using smbpasswd as the backend, and Samba 3 really wants to have an LDAP backend. We use LDAP for our UNIX accounts, so we just needed to add an ObjectClass for Samba. At least that was the theory. If it wasn't done correctly, the Windows desktops and servers would loose their domain association and everything would come to a stop. This is because Windows domains include a machine account for each machine, which Windows is courtious enough to change the password for periodically. If you don't stop samba on the servers while doing the update, a machine password can be changed while doing the conversion and the mismatch when restarting causes the Windows machine to not be able to communicate with the domain.
This upgrade had been attempted twice before, and despite debugging in a test environment things went poorly and had to be backed out. Due to the previous problems and the need to move forward, it was decided that there was going to be no going back.
The SysAdmin who led the previous attempts at Samba 3 is a full time student at UF and part time with us, so coordinating with him is tricky. Previously he had said everything was essentially ready except for migrating user and machine accounts, which had to be done at the time of the cutover. When he came in around 5pm after classes, he told our team lead what needed to be done before the cutover at 7, then left for some student org meeting and would return after the meeting. It turned out we also needed to write some config files before we could to the upgrade. WTF? #1.
Much much later we had config files and an LDAP update ready. The new configs and LDAP were loaded, and... Success! Mostly anyway. Users were able to log in, Windows machines kept their domain trusts, we just couldn't add new machines to the domain.
We were using Samba 3.0.10, which was the version which shipped with the version of RedHat we were using on the PDC. Prior to 3.0.11, you have to use an account which maps to UNIX UID 0 to add a Windows system to the domain. No problem, a UID 0 account is created, password set, and... No join. Not only no join, but the account had it's sambaLMPassword and sambaNTPassword deleted and the "Disabled" flag added to it's sambaAcctFlags. WTF? says we. Passwords are put back, the account unlocked, and one join later... same thing.
Much muttering, experimenting, and discussing later, the decision was made to try the newest version of Samba 3. It was downloaded and installed. We restart and... nothing works. Cursing ensues. The new Samba is backed out, LDAP restored (from a slave LDAP server) to a version which had been working with Samba 3.0.10, and everything restarted. Things seem to work, other than the machine add script. Things are declared good enough, and we head home at midnight.
Part 2 later...
Anyway, part one is to update our Samba servers to version 3 because AD won't form a domain trust with earlier versions. Last night was the night we updated the Primary Domain Controller. Since I don't know much about Samba, I was mostly along for the ride.
Of course, it's never that simple. Under Samba 2, we were using smbpasswd as the backend, and Samba 3 really wants to have an LDAP backend. We use LDAP for our UNIX accounts, so we just needed to add an ObjectClass for Samba. At least that was the theory. If it wasn't done correctly, the Windows desktops and servers would loose their domain association and everything would come to a stop. This is because Windows domains include a machine account for each machine, which Windows is courtious enough to change the password for periodically. If you don't stop samba on the servers while doing the update, a machine password can be changed while doing the conversion and the mismatch when restarting causes the Windows machine to not be able to communicate with the domain.
This upgrade had been attempted twice before, and despite debugging in a test environment things went poorly and had to be backed out. Due to the previous problems and the need to move forward, it was decided that there was going to be no going back.
The SysAdmin who led the previous attempts at Samba 3 is a full time student at UF and part time with us, so coordinating with him is tricky. Previously he had said everything was essentially ready except for migrating user and machine accounts, which had to be done at the time of the cutover. When he came in around 5pm after classes, he told our team lead what needed to be done before the cutover at 7, then left for some student org meeting and would return after the meeting. It turned out we also needed to write some config files before we could to the upgrade. WTF? #1.
Much much later we had config files and an LDAP update ready. The new configs and LDAP were loaded, and... Success! Mostly anyway. Users were able to log in, Windows machines kept their domain trusts, we just couldn't add new machines to the domain.
We were using Samba 3.0.10, which was the version which shipped with the version of RedHat we were using on the PDC. Prior to 3.0.11, you have to use an account which maps to UNIX UID 0 to add a Windows system to the domain. No problem, a UID 0 account is created, password set, and... No join. Not only no join, but the
Much muttering, experimenting, and discussing later, the decision was made to try the newest version of Samba 3. It was downloaded and installed. We restart and... nothing works. Cursing ensues. The new Samba is backed out, LDAP restored (from a slave LDAP server) to a version which had been working with Samba 3.0.10, and everything restarted. Things seem to work, other than the machine add script. Things are declared good enough, and we head home at midnight.
Part 2 later...
Subscribe to:
Posts (Atom)
