A development SAP HANA database in the S/4HANA landscape I support stopped working because its log volume, /hana/log, reached 100%. Nobody had the SYSTEM password, so the usual fixes weren't available. This case study explains the situation, the decisions I made, what I did, and what I changed afterwards.

At a glance

Details

Type of work

Practical work. I handled this incident myself.

Environment

SAP S/4HANA on SAP HANA 2.0 (system database and tenant), SUSE Linux. Development system of a DEV / QAS / PRD landscape.

Tools

<sid>adm shell, HDB, the name server's emergency password reset, HANA Studio, SQL

SAP references

SAP Note 1679938 (disk full event on the log volume), SAP's documented SYSTEM password reset, ALTER SYSTEM RECLAIM LOG

Step-by-step guide

SAP HANA log volume full: the fix step by step

Hostnames, IP addresses, SIDs and customer details are left out on purpose.

The situation

The database had stopped accepting changes. When /hana/log is full, HANA can't write redo log entries, so anything that changes data, including most administration, waits or fails.

Three things made this harder than a normal "disk full" ticket:

  1. Log backups weren't freeing space. In log mode normal, HANA keeps every log segment until it has been backed up. The segments were piling up because the log backups weren't completing.

  2. Nobody knew the SYSTEM password. The customer didn't have the SYSTEMDB credentials, so I couldn't log on to the database to investigate or to change anything.

  3. Administrative actions were blocked, because the database couldn't write to its log.

It was a DEV system, so there was no production outage. But development work on the system had stopped, and the same misconfiguration in QAS or PRD would have caused a real outage.

The decisions that mattered

Most of the work in this incident was deciding what not to do.

Don't delete log segments. The fastest way to free space is to delete logsegment_*.dat files, and it's the wrong one. Log segments that HANA still needs for restart or recovery can't be recreated, and deleting them by hand can leave a database that won't start or can't be recovered. I ruled that out first.

Get database access back the documented way. Without the SYSTEM password I couldn't check the backup catalog, the log segment states or the backup configuration. SAP documents an emergency reset of the SYSTEM user's password in the system database: stop the database as <sid>adm and start the name server in a special reset mode. It fits this situation: there were no working administrator credentials for the system database. I've written up the procedure in Resetting the HANA SYSTEM user password.

Free space temporarily, without losing anything. HANA needed some free space to start and run its log backups. SAP Note 1679938 describes the emergency approach for a full log volume: move a log segment that is already backed up to another file system, and leave a symbolic link in its place so HANA still finds the file. That frees space without removing anything HANA might need.

Fix the cause, not only the symptom. Freeing space buys time. The space only comes back for good once log backups work again and HANA releases the backed-up segments itself.

What I did

  1. Stopped HANA cleanly as <sid>adm.

  2. Reset the SYSTEM user's password with the name server's emergency mode, following SAP's documented procedure, and got access to the system database back.

  3. Moved one large, already backed-up log segment to another file system and replaced it with a symbolic link, as SAP Note 1679938 describes.

  4. Started HANA and confirmed that log backups were running again.

  5. Ran ALTER SYSTEM RECLAIM LOG in the system database and in the tenant, so the freed segments were removed and the space returned to the file system.

  6. Put the moved segment back in order once HANA had released it, and confirmed a clean startup.

The commands for each step are in the step-by-step guide, and RECLAIM LOG is explained in ALTER SYSTEM RECLAIM LOG.

Result

  • Log space was reclaimed and the DEV system was stable again.

  • No log file was deleted by hand.

  • The root cause was identified: the log backup configuration. Log backups weren't completing, so every segment stayed on disk.

What I changed afterwards

After this incident I check these on every system I look after:

Check

Why

log_mode = normal and automatic log backup enabled

otherwise point-in-time recovery isn't possible, or segments are never released

the log backup destination has free space and is monitored

a full backup destination stops log backups, which then fills /hana/log

recent log backups in the backup catalog are successful

a failing log backup is the early warning

an alert on log volume usage well before 100%

there's time to act before the database stops

someone in the team can access SYSTEMDB with a named admin user

so an emergency reset isn't needed again

The last point matters as much as the technical ones. The emergency reset worked, but it needs a full stop of the database. A named user administrator in SYSTEMDB would have avoided that downtime.

What I learned

  • Read the state before acting. The segment states and the backup catalog told me why the log was full. Without them I would have been guessing.

  • Use SAP's documented emergency paths. SAP Note 1679938 and the documented password reset exist for exactly this situation, and they keep the database recoverable.

  • DEV incidents are rehearsals. The same log backup misconfiguration in production would have meant a production outage. Fixing it in DEV, and writing it down, is cheap practice.

  • Write the procedure down while it's fresh. My notes from this incident became the guides linked below.