rjuju's home

Extracting SQL from WAL? (part 2)

2023-12-20T03:04:10+00:00

In the previous article of this series, we saw how to extract WAL records related to the exact SQL commands we want, INSERTs on heap tables, and what the structure of those records was. In this article we will focus on the heap specific information contained in those records and how to extract SQL queries from them.

INSERT data

At the end of the previous article, we could locate the various xl_heap_insert records from the WAL stream. From there, we extracted some metadata about the file’s physical location (tablespace oid, database oid and relation filenode among other things) and the data that was inserted itself.

As a reminder, here’s an extract of the code responsible for generating the WAL records for an INSERT, in the heap_insert() function, focusing on the interesting data:

void
heap_insert(Relation relation, HeapTuple tup, CommandId cid,
			int options, BulkInsertState bistate)
{
[...]
		xl_heap_header xlhdr;
[...]
		xlhdr.t_infomask2 = heaptup->t_data->t_infomask2;
		xlhdr.t_infomask = heaptup->t_data->t_infomask;
		xlhdr.t_hoff = heaptup->t_data->t_hoff;
[...]
		XLogRegisterBuffer(0, buffer, REGBUF_STANDARD | bufflags);
		XLogRegisterBufData(0, (char *) &xlhdr, SizeOfHeapHeader);
		/* PG73FORMAT: write bitmap [+ padding] [+ oid] + data */
		XLogRegisterBufData(0,
							(char *) heaptup->t_data + SizeofHeapTupleHeader,
							heaptup->t_len - SizeofHeapTupleHeader);
[...]

2 entries are inserted: an xl_heap_header which contains some metadata about the tuple, extracted from the tuple header, and the data part of a HeapTuple. Let’s look at those in details.

Page layout

First of all, let’s quickly see how postgres stores tables and indexes on disk. I will only cover those basics that will be helpful for the rest of the article. If you want to dig more into this topic, there are a tons of resource available. You can refer to this entry point in the code, and I otherwise recommend looking at the section about it in “The internals of postgres” website.

A good general introduction is the documentation, which comes with a diagram of the layout that I include here:

Each tuple and index piece of data that postgres stores on disk is stored into a Page, which is by default 8kB. Each page starts with a header that contains some metadata about the page and ends with an optional “special area”, which can contain additional information specific to the component of postgres that will use this page.

In between is the actual data. The beginning of the data part is an array of ItemId, in ascending order, and the end of the data part are the items themselves (which will be the tuples in case of heap table pages), stored in the reverse order from the ItemId. Unless the page is totally full, there will be an empty space between the last ItemId and the first item (the pd_lower and pd_upper offset in the Page metadata).

Here’s the ItemId definition:

typedef struct ItemIdData
{
	unsigned lp_off:15,  /* offset to tuple (from start of page) */
		 lp_flags:2, /* state of line pointer, see below */
		 lp_len:15;  /* byte length of tuple */
} ItemIdData;

As you can see it holds the location of the item in the page, minimal metadata and the length of the item.

HeapTuple

The largest part stored in the record is the tuple itself. As the historic and default access method to store tuple is called heap, the struct that holds the tuple is called HeapTuple. Any custom Table Access Method can use a different struct to store what it needs for its specific implementation, but it will then also use a custom resource manager to generate specific WAL records.

Here’s the definition of a HeapTuple:

typedef struct HeapTupleData
{
	uint32		t_len;		/* length of *t_data */
	ItemPointerData t_self;		/* SelfItemPointer */
	Oid		t_tableOid;	/* table the tuple came from */
#define FIELDNO_HEAPTUPLEDATA_DATA 3
	HeapTupleHeader t_data;		/* -> tuple header and data */
} HeapTupleData;

It starts with some metadata, which isn’t stored on disk but generated or retrieved from somewhere else when the struct is read from disk. Indeed, there wouldn’t be much value storing the relation’s oid for each tuple on disk. The length of the tuple is stored on disk, as it’s a necessary piece of information, and is retrieved from the associated ItemId the we saw just before.

After that follows the “real” data, which is what is stored in the item part of the Page. It’s again split in 2 parts: the tuple header, which I will cover a bit later, and the tuple data.

The tuple data is the physical on-disk representation of the tuple. It was designed to be as space efficient as possible, so accessing individual fields is a bit complex, and CPU intensive. Let’s the most important part of this design. First, the tuple data is defined like that:

struct HeapTupleHeaderData
{
[...]
	/* ^ - 23 bytes - ^ */

#define FIELDNO_HEAPTUPLEHEADERDATA_BITS 5
	bits8		t_bits[FLEXIBLE_ARRAY_MEMBER];	/* bitmap of NULLs */

	/* MORE DATA FOLLOWS AT END OF STRUCT */
};

You probably know or heard that in postgres, NULL attributes don’t use any storage. Indeed, if an attribute is NULL there won’t be anything in the “data section”, and the bit for its attribute number in the t_bit bitmap will be set.

Then, a lot of data types have a variable size (which is internally referred as varlena). So, to save space postgres doesn’t store the offset of each attributes in the HeapTuple and just stores them next to each other (according to the datatype alignment rules) in a big chunk of memory.

This is indeed efficient, but unless your tuple only contains non-null fixed-sized attribute, the only way to access a specific attribute is to read all the previous ones, skip the NULL attribute and compute the position of the next one reading the length of variable datatype. This process is called tuple deforming, it takes a tuple in input and outputs two arrays: one with the datums and one with the null references, all indexed by the attribute number (0 based). The opposite operation (transform a tuple of datum and a tuple of nulls in a tuple) is unsurprisingly called tuple forming. If you want to read a bit more about those operations, the underlying functions are called heap_deform_tuple() and heap_form_tuple().

Note that tuple deforming is one of the operations that can be JITted, and there are some optimisations on the tuple deforming operation. Postgres supports “partial” deforming and will avoid deforming the full tuple when possible, stopping at the last attribute that the query is referencing, and will cache the offset of the latest attribute that has been deformed. But that can only help to some extent, so it’s always a good idea to mark columns as NOT NULL when possible, put all the columns with fixed-length attributes at the beginning of the tuples (with the NOT NULL first), ideally grouped by alignment size to avoid wasting a few bits, and put the most frequently accessed columns of variable length datatype next. All of that will help speeding up tuple deforming as much as possible.

Tuple header

The first part of the stored data is an xl_heap_header struct. It’s just a shorter version of the real tuple header that only contains some part of it, the rest of the header being available elsewhere in the WAL record or just not needed otherwise. Doing it this way can save a few bytes for each insert in the WAL, which is always a good thing. Its definition is:

typedef struct xl_heap_header
{
	uint16		t_infomask2;
	uint16		t_infomask;
	uint8		t_hoff;
} xl_heap_header;

t_infomask2 and t_infomask2 are two bitmaps that contain information about the tuple. You may have heard about hint bits, those two fields contains the tuple-level hint bits.

Let’s look at their details htup_details.c

struct HeapTupleHeaderData
{
[...]
	/* Fields below here must match MinimalTupleData! */

#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK2 2
	uint16		t_infomask2;	/* number of attributes + various flags */

#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK 3
	uint16		t_infomask;		/* various flag bits, see below */

#define FIELDNO_HEAPTUPLEHEADERDATA_HOFF 4
	uint8		t_hoff;			/* sizeof header incl. bitmap, padding */

	/* ^ - 23 bytes - ^ */
[...]
}

 * information stored in t_infomask2:
 */
#define HEAP_NATTS_MASK			0x07FF	/* 11 bits for number of attributes */
/* bits 0x1800 are available */
#define HEAP_KEYS_UPDATED		0x2000	/* tuple was updated and key cols
										 * modified, or tuple deleted */
#define HEAP_HOT_UPDATED		0x4000	/* tuple was HOT-updated */
#define HEAP_ONLY_TUPLE			0x8000	/* this is heap-only tuple */

#define HEAP2_XACT_MASK			0xE000	/* visibility-related bits */
[...]
 * information stored in t_infomask:
 */
#define HEAP_HASNULL			0x0001	/* has null attribute(s) */
#define HEAP_HASVARWIDTH		0x0002	/* has variable-width attribute(s) */
[...]
#define HEAP_XMIN_COMMITTED		0x0100	/* t_xmin committed */
#define HEAP_XMIN_INVALID		0x0200	/* t_xmin invalid/aborted */
#define HEAP_XMIN_FROZEN		(HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID)
#define HEAP_XMAX_COMMITTED		0x0400	/* t_xmax committed */
#define HEAP_XMAX_INVALID		0x0800	/* t_xmax invalid/aborted */
[...]

We can see a few bits useful for the tuple deforming. For instance, we see that 11 bits of t_infomask2 are used to store the actual number of attributes stored in this tuple. Adding a new column in a table doesn’t always require a full table rewrite, and in that case those bits are critical to know when to stop looking for additional attributes when accessing tuples stored before the column was added. There’s also information on whether the tuple contains any NULL or variable-length datatype attribute. The rest of the hint bits are a clever use of the available space to handle various SQL operations, MVCC rules, HOT updates and other low level optimisations.

Tuple descriptors

Now that we covered some internals of the HeapTuple, it seems much easier to reach our goal: transform the INSERT WAL records into plain SQL statements. We know that we just have to deform the tuples to retrieve the values and the NULL attributes, generating the SQL statements around isn’t hard. But here comes the second reason why we need a proper data directory to do so, and why the lack of DDL is important.

As you probably guessed by now, one critical piece of information needed for the tuple deforming operation is the table structure declaration. Indeed, the HeapTuple is just a big chunk of memory, and without the list of columns, data types, and the types details, it’s impossible to interpret those. If your model doesn’t change too much it’s probably possible to do without and instead generate some kind of mapping manually based on what you know about the history of the instance. Be careful if you go this way, any discrepancy between the original and generated data types can lead to bogus output in the best case, or crashing your whole instance. But in my case I had the guarantee that no DDL happened since the incident, and the other data directory available so I could just rely on it.

Postgres handles the table structure declaration using another struct, called TupleDesc, for tuple descriptor. Its definition is:

typedef struct TupleDescData
{
	int	     natts;	/* number of attributes in the tuple */
	Oid	     tdtypeid;	/* composite type ID for tuple type */
	int32	     tdtypmod;	/* typmod for tuple type */
	int	     tdrefcount;/* reference count, or -1 if not counting */
	TupleConstr *constr;	/* constraints, or NULL if none */
	/* attrs[N] is the description of Attribute Number N+1 */
	FormData_pg_attribute attrs[FLEXIBLE_ARRAY_MEMBER];
} TupleDescData;

In our case the most interesting members are the number of attributes (natts) and the array of pg_attribute records (attrs). Those are also useful for the SQL generation part, as we can retrieve the columns from it. Note also that postgres will generate a TupleDesc automatically when you internally open a relation.

Let’s recapitulate. We have the record data, the filename contains the physical file location information that we can use to retrieve the actual relation, we know how to get the tuple descriptor for this relation and we can use it to deform the tuple and get the values from it. We have almost everything we need to generate the SQL queries.

The only remaining detail is that the values we get from the tuple deforming operation are in their physical representation, and we need to emit their textual representation. Again, that’s not a problem as each data type has a dedicated function for that, called type output function, available in pg_type.typoutput.

Extracting SQL from the INSERT records

Now is time for the fun part where we just need to put everything together to finish the project!

I chose to write it as an extension to be able to add and remove it easily from a production server. I also chose to minimize the amount of C code and rely on plpgsql functions when possible. It’s faster to write and plpgsql is also way safer.

I only wrote a single pg_decode_record() C function, that takes as input a record as a bytea, the tablespace oid and the relation filenode and emits the underlying SQL query. I wrote an extra pg_decode_all_records() function in plpgsql that uses existing pg_ls_dir() and pg_read_binary_file() to retrieve the files and record, and split_part() to extract the metadata from the filename.

I’m attaching the resulting extension to this article so you can see the whole implementation and adapt it if needed, and will just quickly describe the main parts here as we already covered the underlying elements. I’m also only showing here a simplified version to avoid too many implementation details.

First, I look for a matching relation oid in the pg_class catalog for the given tablespace and relfilenode, open the found relation with the weakest lock possible, make a copy of the tuple descriptor and start generating the SQL query with the qualified relation name. As for normal application, you need to make sure that the identifiers are properly quoted to generate working queries:

PGDLLEXPORT Datum
pg_decode_record(PG_FUNCTION_ARGS)
{
    bytea  *record = PG_GETARG_BYTEA_PP(0);
    Oid	spc = PG_GETARG_OID(1);
    Oid	relfilenode = PG_GETARG_OID(2);

    /* Get the relation oid from the tablespace oid and relfilenode */
    relid = get_spc_relnumber_relid(spcOid, relNumber);

    relation = table_open(relid, AccessShareLock);
    tupdesc = CreateTupleDescCopy(RelationGetDescr(relation));

    /* Start generating the SQL query */
    initStringInfo(buf);
    appendStringInfo(buf, "INSERT INTO %s.%s",
    		 quote_identifier(get_namespace_name(RelationGetNamespace(relation))),
    		 quote_identifier(RelationGetRelationName(relation)));

The next part extracts the data from the record and generate a HeapTuple with just enough information to be correctly deformed:

    /* mimic heap_xlog_insert */
    data = VARDATA(record);
    datalen = VARSIZE_ANY(record);
[...]
    htup = &tbuf.hdr;
[...]
    htup->t_hoff = xlhdr.t_hoff;

    /* build a fake tuple with the bare minimum to deform it */
    tuple = (HeapTuple) palloc0(HEAPTUPLESIZE + VARSIZE_ANY(record));
    tuple->t_data = htup;
    tuple->t_len = VARSIZE_ANY(record);
    ItemPointerSetInvalid(&(tuple->t_self));
    tuple->t_tableOid = relid;

For the next step, we just need to allocate the 2 arrays needed for the deforming and call heap_deform_tuple():

    values = palloc0(sizeof(Datum) * tupdesc->natts);
    isnull = palloc0(sizeof(bool) * tupdesc->natts);
    heap_deform_tuple(tuple, tupdesc, values, isnull);

Now that we have all the elements, we just need to iterate over the list of columns in the tuple descriptor, output a NULL if needed, otherwise find the type output function, call it for our value, and output it in the query after escaping it:

    /* append the values */
    appendStringInfoString(buf, " VALUES (");
    for (i = 0; i < tupdesc->natts; i++)
    {
    	char	   *value = NULL;
    	Oid			typoutput;
    	bool		typisvarlena;

    	if (i > 0)
    		appendStringInfoString(buf, ", ");

    	if (isnull[i])
    	{
    		appendStringInfoString(buf, "NULL");
    		continue;
    	}

    	getTypeOutputInfo(TupleDescAttr(tupdesc, i)->atttypid,
    					  &typoutput, &typisvarlena);

    	value = OidOutputFunctionCall(typoutput, values[i]);
    	value = quote_literal_cstr(value);

    	appendStringInfo(buf, "%s", value);

    	pfree(value);
    }
    appendStringInfoString(buf, ");");

Once done, we just need to properly close the relation and return the generated query to the caller:

	table_close(relation, NoLock);

	PG_RETURN_TEXT_P(cstring_to_text(buf.data));
}

And that’s all you need for the basic scenario! The real implementation has a bit more code for various other cases, like very basic TOAST table support, but is still unlikely to correctly handle any weird corner cases that can happen in the wild.

Basic usage

We can finally see the result of all the hard work in this article and the previous one! I will be using a simple scenario, first saving the current WAL position to only keep the records generated afterwards, then removing all the data from the table (without changing its relfilenode) to make sure that we don’t read anything from the table itself.

-- Get the current WAL location
rjuju =# SELECT pg_current_wal_lsn();
 pg_current_wal_lsn
--------------------
 F/46349E80
(1 row)

rjuju=# CREATE EXTENSION pg_decode_record;
CREATE EXTENSION

rjuju=# CREATE TABLE decode_record(id integer, val text storage external);
CREATE TABLE

rjuju=# INSERT INTO decode_record
  SELECT 1, 'simple test';
INSERT 0 1

-- Force a full-page write
rjuju=# CHECKPOINT;
CHECKPOINT

rjuju=# INSERT INTO decode_record
  SELECT 2, 'full-page write';
INSERT 0 1

rjuju=# INSERT INTO decode_record
  SELECT 3, 'a bit big '||string_agg(random()::text, ' ') FROM generate_series(1, 10);
INSERT 0 1

rjuju=# INSERT INTO decode_record
  SELECT 4, 'way bigger '||string_agg(random()::text, ' ') FROM generate_series(1, 120);
INSERT 0 1

-- Check the heap table size and underlying TOAST table size
rjuju=# SELECT oid::regclass::text, pg_size_pretty(pg_relation_size(oid)),
  reltoastrelid::regclass::text, pg_size_pretty(pg_relation_size(reltoastrelid))
  FROM pg_class
  WHERE relname = 'decode_record';
      oid      | pg_size_pretty |      reltoastrelid      | pg_size_pretty
---------------+----------------+-------------------------+----------------
 decode_record | 8192 bytes     | pg_toast.pg_toast_66731 | 8192 bytes
(1 row)

rjuju=# DELETE FROM decode_record;
DELETE 4

-- Make sure we remove all records and physically empty the tables
rjuju=# VACUUM decode_record;
VACUUM

rjuju=# SELECT oid::regclass::text, pg_size_pretty(pg_relation_size(oid)),
  reltoastrelid::regclass::text, pg_size_pretty(pg_relation_size(reltoastrelid))
  FROM pg_class
  WHERE relname = 'decode_record';
      oid      | pg_size_pretty |      reltoastrelid      | pg_size_pretty
---------------+----------------+-------------------------+----------------
 decode_record | 0 bytes        | pg_toast.pg_toast_66737 | 0 bytes
(1 row)

Ok, we should have a few records generated in the WAL corresponding to data we definitely lost in the table. Let’s extract the INSERT records using the custom pg_waldump we created in the previous article:

$ mkdir -p /tmp/pg_decode_record
$ pg_waldump --start "F/46349E80" --save-records /tmp/pg_decode_record
[...]
$ ls -l /tmp/pg_decode_record
0000000F-46367520.1663.16384.66743.0_main
0000000F-46367660.1663.16384.66743.0_main
0000000F-46367738.1663.16384.66743.0_main
0000000F-46367868.1663.16384.66746.0_main
0000000F-46368130.1663.16384.66746.0_main
0000000F-46368300.1663.16384.66743.0_main

You might wonder why there are 6 records extracted while we only inserted 4 rows. That’s because the last record was big enough to be TOASTed using 2 chunks, and as far as the WAL are concerned that’s 3 separate INSERTs in 2 different tables. Let’s see that in detail using the extension to decode the records (truncating the output as some rows are quite big):

rjuju=# SELECT substr(v, 1, 95)
    FROM pg_decode_all_records('/tmp/pg_decode_records') f(v);
                                          substr
-------------------------------------------------------------------------------------------
 INSERT INTO public.decode_record (id, val) VALUES ('1', 'simple test');
 INSERT INTO public.decode_record (id, val) VALUES ('2', 'full-page write');
 INSERT INTO public.decode_record (id, val) VALUES ('3', 'a bit big 0.5356172842583808 0.3...'
 INSERT INTO pg_toast.pg_toast_66810 VALUES ('66815', '0', E'\\x7761792062696767657220302e...'
 INSERT INTO pg_toast.pg_toast_66810 VALUES ('66815', '1', E'\\x3337383137353120302e303439...'
 INSERT INTO public.decode_record (id, val) VALUES ('4', /* toast pointer 66815 */);
(6 rows)

(note: I slightly edited the output to make it smaller and have correct syntax highlighting, the real extension will emit the real table name in a comment in case of INSERT in a TOAST table)

We see the first normal records properly decoded, whether they’re in a full-page image or not. The last record is indeed split into 3 different INSERTs, 2 in the TOAST table and 1 in the heap table.

As I mentioned earlier I only added very minimal support for TOAST tables, as I didn’t have any information about the customer tables and whether they would hit that case or not, or how often. The last insert isn’t a valid statement as the 2nd value is missing, but we can manually extract the value from the INSERT statements in the TOAST table and therefore fix the normal INSERT. For instance, using the first few bytes that we can see in the first chunk:

rjuju=# SELECT encode(E'\\x7761792062696767657220302e', 'escape');
-[ RECORD 1 ]---------
encode | way bigger 0.

The data is there, it just needs a bit of manual processing to get it.

To be totally fair, I also cheated a bit in that example by making sure that the data will be TOASTed but not compressed, so it’s very easy to manually retrieve the raw value from the extra INSERTs in the TOAST tables. It wouldn’t be very hard to have all of that working transparently, but I simply didn’t have the need. If you’re interested in that, I’d recommend looking at the detoast_attr() function in src/backend/access/common/detoast.c and all underlying code to see how you can manually decompress data. You would then only need to store the detoasted (and potentially decompressed) value referenced by the toast’s chunk_id locally, and emit it in the query instead of the currently emitted comment.

Conclusion

I hope you enjoyed those two articles and learned a bit about the WAL infrastructure and the way pages and tuples work internally.

If you missed it in the article, here is the link for the full extension.

I want to emphasize again that all the code I showed here is only a quick proof of concept that’s thought for one narrow use case, and it should be used with care. My goal here wasn’t to show state of the art code but rather show one possible way to quickly come up with a plan to salvage data in case of production incident. If you’re unfortunately confronted to a similar problem, or some major other accident I hope you will find some valuable resources and a starting point to come up with your own dedicated solution!

Extracting SQL from WAL? (part 2) was originally published by Julien Rouhaud at rjuju's home on December 20, 2023.

Extracting SQL from WAL? (part 1)

2023-12-06T03:04:10+00:00

Is it actually possible to extract SQL commands from WAL generated in “replica” wal_level?

The answer is usually no, the “logical” wal_level exists for a reason after all, and you shouldn’t expect some kind of miracle here.

But in this series of articles you will see that if some conditions are met you can still manage to extract some information, and how to do it. This first article focuses on the WAL records and how to extract the ones you want, while the next one will show how to try to extract the information contained in those records.

Some context

This article is based of some work I did a few months ago to help a customer recover some data after an incident. It’s not a perfect solution and mostly a set of quick hacks I did to come up with something able to retrieve data in a few hours of work only, but I hope sharing details about it and some methodology can be helpful if you ever get in a similar situation. You will probably need to adapt it to your needs, with yet other hacks, but it should give you a good start. It can otherwise be of some interest if you want to know a bit more about the WAL records internals and some associated infrastructure.

The incident

Due to a series of unfortunate events, one of their HA clusters ended in a split-brain situation for a some time before being reinitialised, which entirely removed one of the data directory. After that, only the WALs that were were generated on that instance were available, those being in “replica” wal_level, and nothing else.

One possibility to try recover the data would be to restore a physical backup, if any, replay archived WALs until the last transaction before the removed node is promoted (assuming those are still available) and then replay the WALs generated on that newly promoted node. Once there you still need to look at each row of each table of each database and compare it to yet another instance restore from the same backup to approximately the same time as this one. That’s clearly not ideal as it will likely require many days or even weeks of tedious hard work to do so, and will consume a lot of resources along the way. Is there a way to do better?

After a quick discussion, it turned out that there were a few elements that made some recovery from the WALs themselves possible (more on why later):

One of the data directories was still available
The customer guaranteed that no DDL happened since the incident
Only INSERTs happened during the split-brain

WALs & Physical replication

As you probably know, postgres physical replication works by sending an exact copy of the modified binary raw data to the various standby servers, in a continuous stream of WAL records. As a consequence, those records don’t really know much about the database objects they reference, and nothing about the SQL queries that generated them. So what do they really contain? Let’s see what’s inside the WAL records generated for an INSERT into a normal heap relation.

WAL records

First of all, you have to know that the WAL records are split into Resource Managers (declared in src/include/access/rmgrlist.h), each being responsible for a specific part of postgres (heap tables, indexes, vauum…). They’re identified by a numeric identifier and often referred to as a rmid, for //resource manager identifier//.

Each of those resource managers can handle various operations, which are internally called opcodes. Here we’re interested in the WAL records generated while operating on standard heap tables, and especially during INSERTs. This resource manager is a bit particular as it’s split into 2 different rmid: RM_HEAP_ID and RM_HEAP2_ID. This is only an implementation details, as each resource manager can only handle a limited number of opcodes, everything is the same otherwise.

If you’re curious, here’s the definition of the main WAL record in the source code and a bit of details on the exact layout in the files:

/*
 * The overall layout of an XLOG record is:
 *		Fixed-size header (XLogRecord struct)
 *		XLogRecordBlockHeader struct
 *		XLogRecordBlockHeader struct
 *		...
 *		XLogRecordDataHeader[Short|Long] struct
 *		block data
 *		block data
 *		...
 *		main data
 * [...]
 */
typedef struct XLogRecord
{
	uint32		xl_tot_len;		/* total len of entire record */
	TransactionId xl_xid;		/* xact id */
	XLogRecPtr	xl_prev;		/* ptr to previous record in log */
	uint8		xl_info;		/* flag bits, see below */
	RmgrId		xl_rmid;		/* resource manager for this record */
	/* 2 bytes of padding here, initialize to zero */
	pg_crc32c	xl_crc;			/* CRC for this record */

	/* XLogRecordBlockHeaders and XLogRecordDataHeader follow, no padding */

} XLogRecord;

and a block data header:

/*
 * Header info for block data appended to an XLOG record.
 *
 * 'data_length' is the length of the rmgr-specific payload data associated
 * with this block. It does not include the possible full page image, nor
 * XLogRecordBlockHeader struct itself.
 *
 * Note that we don't attempt to align the XLogRecordBlockHeader struct!
 * So, the struct must be copied to aligned local storage before use.
 */
typedef struct XLogRecordBlockHeader
{
	uint8		id;				/* block reference ID */
	uint8		fork_flags;		/* fork within the relation, and flags */
	uint16		data_length;	/* number of payload bytes (not including page
								 * image) */

	/* If BKPBLOCK_HAS_IMAGE, an XLogRecordBlockImageHeader struct follows */
	/* If BKPBLOCK_SAME_REL is not set, a RelFileLocator follows */
	/* BlockNumber follows */
} XLogRecordBlockHeader;

Everything here is very generic as it’s used by all the resource managers. One important bit though is the mention of a RelFileLocator after the block header if the record contains information about a different relation from the previous block, whatever is was (which is the meaning of BKPBLOCK_SAME_REL). This is of course important information for us.

typedef struct RelFileLocator
{
	Oid			spcOid;			/* tablespace */
	Oid			dbOid;			/* database */
	RelFileNumber relNumber;	/* relation */
} RelFileLocator;

But here’s a first reason why you need a proper data directory to do anything with the WALs: this doesn’t contain the schema name and table name, or even the table oid, but the tablespace oid, database oid and relfilenode, which is what the WAL actually need to identify a physical relation file (which is itself split into multiple files, the exact fork and segment are deduced using other information). So any table rewrite happening since the WAL records were generated (e.g. a VACUUM FULL) and you won’t be able to identify which relation a record is about, unless of course you find a way to map the current relfilenode to the one before the table rewrite.

Heap INSERT WAL records

Now that we saw a bit of the general WAL structures, let’s focus on the data specific to an INSERT. If you’re not familiar really with the internals, one easy way to locate the code related to a specific command is to look at the functions associated to a resource manager. Let’s look at the RM_HEAP_ID information in src/include/access/rmgrlist.h:

/* symbol name, textual name, redo, desc, identify, startup, cleanup, mask, decode */
PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask, heap_decode)

We here have the name of the actual functions responsible for many operations (the exact list will vary depending on the postgres major version, I’m here using the list in postgres 17).

The redo function is the name of the function that applies an RM_HEAP_ID record, the desc functions is the one that emits the info you see in pg_waldump, the identify function returns a string describing the opcode and so on. Let’s look at heap_identify():

const char *
heap_identify(uint8 info)
{
	const char *id = NULL;

	switch (info & ~XLR_INFO_MASK)
	{
		case XLOG_HEAP_INSERT:
			id = "INSERT";
			break;
[...]
	}

	return id;
}

We now know that the opcode we’re interested in is XLOG_HEAP_INSERT. A quick git grep in the tree will lead you to src/backend/access/heap/heapam.c, more precisely the heap_insert function. The interesting bit is located in the “XLOG stuff” block. I will show here an extract focusing on the bit we will need:

void
heap_insert(Relation relation, HeapTuple tup, CommandId cid,
			int options, BulkInsertState bistate)
{
[...]
	/* XLOG stuff */
	if (RelationNeedsWAL(relation))
	{
		xl_heap_insert xlrec;
		xl_heap_header xlhdr;
		XLogRecPtr	recptr;
		Page		page = BufferGetPage(buffer);
		uint8		info = XLOG_HEAP_INSERT;
		int			bufflags = 0;
[...]
		xlrec.offnum = ItemPointerGetOffsetNumber(&heaptup->t_self);
		xlrec.flags = 0;
[...]
		XLogBeginInsert();
		XLogRegisterData((char *) &xlrec, SizeOfHeapInsert);

		xlhdr.t_infomask2 = heaptup->t_data->t_infomask2;
		xlhdr.t_infomask = heaptup->t_data->t_infomask;
		xlhdr.t_hoff = heaptup->t_data->t_hoff;

		/*
		 * note we mark xlhdr as belonging to buffer; if XLogInsert decides to
		 * write the whole page to the xlog, we don't need to store
		 * xl_heap_header in the xlog.
		 */
		XLogRegisterBuffer(0, buffer, REGBUF_STANDARD | bufflags);
		XLogRegisterBufData(0, (char *) &xlhdr, SizeOfHeapHeader);
		/* PG73FORMAT: write bitmap [+ padding] [+ oid] + data */
		XLogRegisterBufData(0,
							(char *) heaptup->t_data + SizeofHeapTupleHeader,
							heaptup->t_len - SizeofHeapTupleHeader);
[...]
		recptr = XLogInsert(RM_HEAP_ID, info);

		PageSetLSN(page, recptr);
	}

We see here that this function is as expected inserting an RM_HEAP_ID record, with an XLOG_HEAP_INSERT opcode. There are 2 data parts associated with this record: the header of the tuple that’s being inserted and the tuple itself.

That’s great! At this point we know how to identify what relation an INSERT is about and the content of that INSERT. Let’s see how to filter those records from the WALs.

Extracting and filtering WAL records

Parsing the postgres WALs isn’t that complicated but still requires to know quite a bit more than what I showed here. Writing such code is possible but wait, don’t we already have a tool shipped with postgres which is designed to do exactly that? Yes there sure is, it’s pg_waldump.

Rather that writing something similar, couldn’t we simply teach pg_waldump to filter the records we’re interested in and save them somewhere so that we can later process them and generate SQL queries? This way we can then also benefit from all options in pg_waldump like specifying the starting and/or ending LSN or filtering a specific resource manager, without the need to worry about most of the WAL implementation details and only focusing on the few functions provided by postgres necessary for our need. Let’s see how to implement that.

The main source file is src/bin/pg_waldump/pg_waldump.c. Skipping most of the unrelated code, we can see that there’s a main loop that takes care of reading each record one by one, optionally filter them and then do something with them depending on how the tool was executed. I will again show an extract to focus on the most relevant part only:

	for (;;)
	{
[...]
		/* try to read the next record */
		record = XLogReadRecord(xlogreader_state, &errormsg);
[...]
		/* apply all specified filters */
		if (config.filter_by_rmgr_enabled &&
			!config.filter_by_rmgr[record->xl_rmid])
			continue;

[...]

		/* perform any per-record work */
		if (!config.quiet)
		{
			if (config.stats == true)
			{
				XLogRecStoreStats(&stats, xlogreader_state);
				stats.endptr = xlogreader_state->EndRecPtr;
			}
			else
				XLogDumpDisplayRecord(&config, xlogreader_state);
		}

		/* save full pages if requested */
		if (config.save_fullpage_path != NULL)
			XLogRecordSaveFPWs(xlogreader_state, config.save_fullpage_path);

		/* check whether we printed enough */
		config.already_displayed_records++;
		if (config.stop_after_records > 0 &&
			config.already_displayed_records >= config.stop_after_records)
			break;
	}

That’s quite simple, pg_waldump read the records one by one until it needs to stop, ignore the records that the users asked to discard and then takes action on the remaining ones. We can see that there’s already an option to save full page images, it definitely looks like we could just add something similar there, but for all records.

First, we will need to provide a way to identify the relation the INSERT is about. That’s the RelFileLocator, and we already know that it can be found just after the XLogRecordBlockHeader. Postgres provides a function to retrieve this information, and a bit more, named XLogRecGetBlockTagExtended(). Here is it’s description:

/*
 * Returns information about the block that a block reference refers to,
 * optionally including the buffer that the block may already be in.
 *
 * If the WAL record contains a block reference with the given ID, *rlocator,
 * *forknum, *blknum and *prefetch_buffer are filled in (if not NULL), and
 * returns true.  Otherwise returns false.
 */
bool
XLogRecGetBlockTagExtended(XLogReaderState *record, uint8 block_id,
						   RelFileLocator *rlocator,
 						   ForkNumber *forknum,
						   BlockNumber *blknum,
						   Buffer *prefetch_buffer)

We need to provide the record - pg_waldump already retrieves it for us - and the block_id. The block_id, or block reference, is simply an offset in the array of data that the WAL records contains. If you look a bit above in this article, you will see that we already know that heap_insert() only uses a hardcoded 0 block_id: this is the first argument in the various XLogRegisterXXX() function calls.

Next we need to retrieve the actual WAL record data, the tuple header and the tuple itself. This one is a bit trickier, as the record can either be found in a simple WAL record or in a full-page record. We need to check for a simple WAL record first. The associated function is XLogRecGetBlockData():

/*
 * Returns the data associated with a block reference, or NULL if there is
 * no data (e.g. because a full-page image was taken instead). The returned
 * pointer points to a MAXALIGNed buffer.
 */
char *
XLogRecGetBlockData(XLogReaderState *record, uint8 block_id, Size *len)

As noted in the comment, if the function returns NULL (and sets len to 0) then the data may be in a full-page image instead (or the data could be missing entirely). If that’s the case we need to retrieve the full-page image, and then locate the tuple the INSERT was about and extract it in the same format as a simple WAL record.

Postgres provides a function to extract the full-page image: RestoreBlockImage():

/*
 * Restore a full-page image from a backup block attached to an XLOG record.
 *
 * Returns true if a full-page image is restored, and false on failure with
 * an error to be consumed by the caller.
 */
bool
RestoreBlockImage(XLogReaderState *record, uint8 block_id, char *page)

which is straightforward to use: just provide the record and the block identifier and you get the full-page image if found. However, there’s no function available to extract a tuple for a full-page image. Indeed postgres can simply overwrite the whole block with the full-page image as it contains the latest version of the block at the time it was generated, but in our case we definitely don’t want to emit an INSERT statement for every already existing tuple in the block!

Fortunately, even when we get a full-page image, our record still contains a //main data area//. If you look up at the heap_insert() function, that’s the call to XLogRegisterData(), and as you see here it contains an xl_heap_insert struct. And the first member of this struct, offnum, is actually the position of the tuple in the page which is exactly what we need!

With all of that, it’s just a matter of accessing the tuple header and tuple at the correct place among all the tuples present in the page, and save as we would way it would be if it were a simple WAL record. If you’re wondering how exactly it should be done, you can always look at how postgres itself does it when it needs to return a specific tuple and adapt that code to your need. The functions responsible for that are heapgetpage() and heapgettup(), located in the src/backend/access/heap/heapam.c file we already mentioned.

We now have the information about the physical file location and the record itself that we will need to transmit to another program to decode it. The best way to do that is to simply save the record as-is in a binary file, and use the file name to transmit the metadata. I chose the following pattern to name the produced files:

LSN.TABLESPACE_OID.DATABASE_OID.RELFILENODE.FORKNAME

It will be trivial for the consumer to parse it and extract the required metadata. One thing to note is that I don’t put the rmid or the opcode here as I’m only emitting the only one I’m interested in and discard everything else. If that’s not your case you should definitely remember to add those in the filename pattern.

Since this requires a bit of code to implement, I won’t detail it here but you can find the full result in the patch for pg_waldump that I’m attaching to this article, which implements this as a new –save-records option.

To conclude, let me also remind you that a compiled version of pg_waldump will only work for a single major postgres version. In my case, I had to work with postgres 11, so you can find the patch for this version here, but if needed I also rebased it again the current commit on the master branch, which can be found here.

What’s next?

This is the end of this first article. We saw some details on the postgres WAL infrastructure, with a full example for the case of a plain INSERT on a heap table. We also learned where to look to find where other WAL records are generated and to see more details about the implementation.

We also checked how pg_waldump is working and how to adapt it for our need, with a provided complete patch for both postgres 11 and the current dev version (postgres 17). Again, I’d like to remind you that all this work is only at a proof-of-concept stage, it’s definitely not polished and I’m sure that are many problems that would need to be fixed. One obvious example of such problem is that we’re saving all INSERT we find in the logs but we don’t check if the transaction they’re in eventually committed. It would be possible to fix that but it would require extraneous code, so as is it’s up to the users to double check that as needed. Overall it was enough to recover the needed data so I didn’t pursue any more work on it.

In the next article we will see some usage of this new –save-records option, and also how to read those records and decode them to generate plain INSERT queries. Stay tuned!

Extracting SQL from WAL? (part 1) was originally published by Julien Rouhaud at rjuju's home on December 06, 2023.

Queryid reporting in plpgsql_check

2020-11-17T02:42:33+00:00

plpgsql_check version 1.14.0 was just released and brings some improvement for performance diagnostic.

Thanks a lot to Pavel Stěhule for the awesome plpgsql_check extension and the help for implementing the queryid reporting in v1.14!

plpgsql_check: static code analysis and more

PostgreSQL supports procedural code for many languages, the most popular one probably being plpgsql.

Even if that language allows you to write raw SQL statements, any function written in that language is still a block box as far as PostgreSQL is concerned, which means that PostgreSQL won’t perform a lot of checks to verify code quality, typo or any other problem related to code development. That’s where plpgsql_check extension comes into play.

If you write any plpgsql code, this extension will be your best friend, as it brings so many cool features. The major feature is static code analysis, which can detect many bugs, security / SQL inject issue and even possible performance issue by detecting implicit casts that could prevent PostgreSQL from using indexes and much more.

It also brings a simple, but yet very useful, code profiler.

How to track down performance issue in plpgsql code?

As I mentioned above, plpgsql code is a black box as far as PostgreSQL is concerned. The direct consequence is that the performance diagnostic possibilities are quite limited.

Using core PostgreSQL, the only option is using pg_stat_user_functions (which requires track_functions to be set to pl or all). It’ll show the number of time each function has been called, and how long the execution took including and excluding nested functions. Unfortunately, this view can only help you track down which function is slow, but not why, as you don’t get any per-instruction metric.

You can somehow work around that limitation using the contrib extensions pg_stat_statements. This extensions is one of the most popular one as far as performance diagnostic is concerned, and gives you a lot of data on query performance (including planning counters and WAL counters since PostgreSQL 13).

The only problem is that it can be quite tricky to match pg_stat_statements entries with your plpgsql code, as there’s way to directly identify which queries are run inside your plpgsql code.

plpgsql_check code profiler

Another alternative is to use a plpgsql code profiler. There are multiple extensions that bring this feature, and I personally chose plpgsql_check, as it perfectly suited my need: simple to setup and use, all performance information I needed and possibility to use it either in a per-connection base or globally when configuration the extension in shared_preload_libraries. Thanks to this profiler, you can finally get performance metrics at the statement level inside plpgsql code:

total execution time, that is the cumulated execution time for all the statements in the source code line
average execution time, that is the total execution time divided by the number of statements in the source code line
maximum execution time, per statement
number of rows processed, per statement

With those information, it becomes quite easy to track down the slow part of your functions. Here’s a simplistic example:

=# SELECT lineno, cmds_on_row, total_time, avg_time, max_time, source
  FROM plpgsql_profiler_function_tb('pltest()');
 lineno | cmds_on_row | total_time | avg_time |     max_time     |                        source
--------+-------------+------------+----------+------------------+-------------------------------------------------------
      1 |      <NULL> |     <NULL> |   <NULL> | <NULL>           |
      2 |      <NULL> |     <NULL> |   <NULL> | <NULL>           | DECLARE
      3 |      <NULL> |     <NULL> |   <NULL> | <NULL>           |     num bigint;
      4 |      <NULL> |     <NULL> |   <NULL> | <NULL>           |     _tbl text = 'pg_class';
      5 |           1 |      0.085 |    0.085 | {0.085}          | BEGIN
      6 |           1 |      0.504 |    0.504 | {0.504}          |     drop table if exists meh;
      7 |           1 |       0.81 |     0.81 | {0.81}           |     CREATE TABLE meh(id integer);
      8 |           1 |      0.362 |    0.362 | {0.362}          |     EXECUTE 'SELECT COUNT(*) FROM ' || _tbl INTO num;
      9 |           2 |    1000.84 |   500.42 | {0.349,1000.491} |     delete from meh; PERFORM pg_sleep(1);
     10 |           1 |          0 |        0 | {0}              |     RETURN num;
     11 |      <NULL> |     <NULL> |   <NULL> | <NULL>           | END;
(11 rows)

In this example, we can see immediately that the slowdown comes from source code line n°9, which has a total execution time of 1s. Using the max_time field, we see that it’s because of the 2nd statements. As we also have the source code available in the view, we can immediately see the problematic query, which here is a simple call to pg_sleep(1).

So far so good. But with less naive example the cause of slow execution might be less obvious, and it could be handy to rely on all the available extensions to get more information: pg_stat_statements for general counters, pg_stat_kcache for CPU and disk usage counters, pg_wait_sampling for wait events and so on.

But how to match the plpgsql statement with entries in those extensions?

Exposing queryid in plpgql_check profiler

Indeed, those extensions identify queries using a query identifier, computed by pg_stat_statements. You could try to manually find the related entry using the query text stored by pg_stat_statements, but it may not always be possible. What if the query is dynamic SQL or using unqualified names?

The solution here is quite simple: since plpgsql_check profiler already show per-statement information, also report the statement’s underlying queryid.

This is now available with version 1.14.0. Using the previous naive example, here’s what we now see:

=# SELECT lineno, max_time, queryids, source
  FROM plpgsql_profiler_function_tb('pltest()');
 lineno |     max_time     |                 queryids                  |                        source
--------+------------------+-------------------------------------------+-------------------------------------------------------
      1 | <NULL>           | <NULL>                                    |
      2 | <NULL>           | <NULL>                                    | DECLARE
      3 | <NULL>           | <NULL>                                    |     num bigint;
      4 | <NULL>           | <NULL>                                    |     _tbl text = 'pg_class';
      5 | {0.085}          | <NULL>                                    | BEGIN
      6 | {0.504}          | {NULL}                                    |     drop table if exists meh;
      7 | {0.81}           | {NULL}                                    |     CREATE TABLE meh(id integer);
      8 | {0.362}          | {-7484655548452190292}                    |     EXECUTE 'SELECT COUNT(*) FROM ' || _tbl INTO num;
      9 | {0.349,1000.491} | {8162364748417812595,6729783856403017864} |     delete from meh; PERFORM pg_sleep(1);
     10 | {0}              | <NULL>                                    |     RETURN num;
     11 | <NULL>           | <NULL>                                    | END;
(11 rows)

You’re now only a JOIN away from matching your plpgsql profile data from your favorite extensions!

Limitations

There are unfortunately some limitations.

Due to pg_stat_statements implementation, queryid for DDL queries is not exposed outside the extension, so plpgsql_check can’t retrieve it.

When using dynamic SQL, there might be many queries involved:

the query text itself will be generated using SQL statement(s)
the parameters, if any, will also be resolved running SQL statement(s)
if the query text depends on some parameters, you can end up with multiple different top level query

plpgsql_check will only report the top level query identifier, and if multiple different queries are generated only the query identifier of the first one will be reported.

Even with those limitations I still hope that this new feature will be helpful.

What’s next?

Due to current plpgsql implementation, when a dynamic SQL statement is executed the query identifier is not visible outside plpgsql itself. It means that retrieving the query identifier in that case is a bit costly, as plpgsql_check has to do some of the work that plpgsql is doing:

generate the final query string
parse the query string
call the parse analysis step (this is where the query identifier is generated)

Of course the query itself won’t be executed or even planned, but those extra steps might add non negligible overhead, especially when the dynamic SQL is executing very short OLTP-style queries.

So plpgsql should be modified to be able to report the query identifier of all statements, whether static or dynamic, so external modules can access the information easily and without any additional overhead. Ideally, this could also be available in plpgsql code using a GET [ CURRENT ] DIAGNOSTICS command, so users can also use it as they need.

Queryid reporting in plpgsql_check was originally published by Julien Rouhaud at rjuju's home on November 17, 2020.

New in pg13: WAL monitoring

2020-04-07T15:46:15+00:00

Write-Ahead Logs is a critical part of PostgreSQL, that ensures data durability. While there are multiple configuration parameters , there was no easy to monitor WAL activity, or what is generating it.

New infrastructure to track WAL activity

commit df3b181499b40523bd6244a4e5eb554acb9020ce
Author: Amit Kapila <akapila@postgresql.org>
Date:   Sat Apr 4 10:02:08 2020 +0530

    Add infrastructure to track WAL usage.

    This allows gathering the WAL generation statistics for each statement
    execution.  The three statistics that we collect are the number of WAL
    records, the number of full page writes and the amount of WAL bytes
    generated.

    This helps the users who have write-intensive workload to see the impact
    of I/O due to WAL.  This further enables us to see approximately what
    percentage of overall WAL is due to full page writes.

    In the future, we can extend this functionality to allow us to compute the
    the exact amount of WAL data due to full page writes.

    This patch in itself is just an infrastructure to compute WAL usage data.
    The upcoming patches will expose this data via explain, auto_explain,
    pg_stat_statements and verbose (auto)vacuum output.

    Author: Kirill Bychik, Julien Rouhaud
    Reviewed-by: Dilip Kumar, Fujii Masao and Amit Kapila
    Discussion: https://postgr.es/m/CAB-hujrP8ZfUkvL5OYETipQwA=e3n7oqHFU=4ZLxWS_Cza3kQQ@mail.gmail.com

With this new infrastructure, each backend will track various information about WAL generation: the number of WAL records, the size of WAL generated and the number of full page images generated. It also makes sure that parallel queries, both DML and utility statements (for now only CREATE INDEX and VACUUM) are correctly handled.

Per-query WAL activity with pg_stat_statements

commit 6b466bf5f2bea0c89fab54eef696bcfc7ecdafd7
Author: Amit Kapila <akapila@postgresql.org>
Date:   Sun Apr 5 07:34:04 2020 +0530

    Allow pg_stat_statements to track WAL usage statistics.

    This commit adds three new columns in pg_stat_statements output to
    display WAL usage statistics added by commit df3b181499.

    This commit doesn't bump the version of pg_stat_statements as the
    same is done for this release in commit 17e0328224.

    Author: Kirill Bychik and Julien Rouhaud
    Reviewed-by: Julien Rouhaud, Fujii Masao, Dilip Kumar and Amit Kapila
    Discussion: https://postgr.es/m/CAB-hujrP8ZfUkvL5OYETipQwA=e3n7oqHFU=4ZLxWS_Cza3kQQ@mail.gmail.com

This basically exposes the mentionned new information about WAL activity in pg_stat_activity, so per (user, database, normalized query). Here is an example:

=# CREATE TABLE t1 (id integer);
CREATE

=# INSERT INTO t1 SELECT 1;
INSERT 0 1

=# UPDATE t1 SET id = 2 WHERE id = 1;
UPDATE 1

=# CHECKPOINT;
CHECKPOINT

=# DELETE FROM t1 WHERE id = 2;
DELETE 1
=# SELECT query, wal_records, wal_bytes, wal_num_fpw
   FROM pg_stat_statements
   WHERE query LIKE 'UPDATE%' OR query LIKE 'DELETE%';
                   query                | wal_records | wal_bytes | wal_num_fpw
-------------------------------------+-------------+-----------+-------------
 DELETE FROM t1 WHERE id = $1        |           1 |       155 |           1
 UPDATE t1 SET id = $1 WHERE id = $2 |           1 |        69 |           0
(2 rows)

I simply inserted a row, updated it and deleted it. Now, looking specifically at the UPDATE and the DELETE, the numbers can be surprising.

When inserting a row, we indeed expect a single WAL record and some WAL bytes for the new row, with some overhead due to internal implementation.

Now, if you’re familiar with PostgreSQL MVCC implementation, you should know that doing a DELETE should only write a transaction id in the xmax field (this documentation page is a good introduction on that subject). So why writing a 4B field (the size of the recotded xmax field), even with some overhead, is writing more than twice the amount of WAL that was required to update a full row? That’s because the DELETE caused a full page write. This is a side effect of performing a CHECKPOINT before the DELETE. To guarantee data consistency (and if full_page_writes parameter isn’t deactivated), any block modified for the first time after a CHECKPOINT completion will be fully logged, rather than logging only the delta.

You’ll also note that the full page didn’t generate 8kB of data as you could expect. This isn’t because of wal_compression, as I didn’t activate it, but because the page is almost empty. Indeed, as an optimization, any “hole” in a page, as long as it’s a standard page, can be safely skipped in the WAL. If you’re curious, this is done in the XLogRecordAssemble() function . Here’s the relevant extract:

static XLogRecData *
XLogRecordAssemble(RmgrId rmid, uint8 info,
				   XLogRecPtr RedoRecPtr, bool doPageWrites,
				   XLogRecPtr *fpw_lsn, int *num_fpw)
{
[...]
		/*
		 * If needs_backup is true or WAL checking is enabled for current
		 * resource manager, log a full-page write for the current block.
		 */
		include_image = needs_backup || (info & XLR_CHECK_CONSISTENCY) != 0;

		if (include_image)
		{
			Page		page = regbuf->page;
			uint16		compressed_len = 0;

			/*
			 * The page needs to be backed up, so calculate its hole length
			 * and offset.
			 */
			if (regbuf->flags & REGBUF_STANDARD)
			{
				/* Assume we can omit data between pd_lower and pd_upper */
				uint16		lower = ((PageHeader) page)->pd_lower;
				uint16		upper = ((PageHeader) page)->pd_upper;

				if (lower >= SizeOfPageHeaderData &&
					upper > lower &&
					upper <= BLCKSZ)
				{
					bimg.hole_offset = lower;
					cbimg.hole_length = upper - lower;
				}
				else
				{
					/* No "hole" to remove */
					bimg.hole_offset = 0;
					cbimg.hole_length = 0;
				}
			}
            [...]

WAL activity in EXPLAIN (and auto_explain)

A new WAL option is available in the EXPLAIN command, and similarly a auto_explain.log_wal for auto_explain, to display the same counters. In TEXT mode, only the non-zero counters are shown, similarly to other counters. For instance:

=# EXPLAIN (ANALYZE, WAL, COSTS OFF) UPDATE t1 SET id = 1 WHERE id = 1;
                           QUERY PLAN
----------------------------------------------------------------
 Update on t1 (actual time=0.181..0.181 rows=0 loops=1)
   WAL:  records=1  bytes=68
   ->  Seq Scan on t1 (actual time=0.074..0.080 rows=1 loops=1)
         Filter: (id = 1)
 Planning Time: 0.274 ms
 Execution Time: 0.381 ms
(6 rows)

WAL activity in autovacuum logs

And finally, if an autovacuum is logging its activity (when reaching the log_autovacuum_min_duration threshold), the same information will be logged. For instance, after inserting 100k records in the same table, deleting half of them and running a CHECKPOINT, here’s the output I get:

LOG:  automatic vacuum of table "rjuju.public.t1": index scans: 0
	pages: 0 removed, 443 remain, 0 skipped due to pins, 0 skipped frozen
	tuples: 50000 removed, 50001 remain, 0 are dead but not yet removable, oldest xmin: 496
	buffer usage: 912 hits, 3 misses, 448 dirtied
	avg read rate: 0.084 MB/s, avg write rate: 12.485 MB/s
	system usage: CPU: user: 0.17 s, system: 0.00 s, elapsed: 0.28 s
	WAL usage: 1330 records, 445 full page writes, 2197104 bytes

This new log output is in my opinion especially important, especially when it comes to anti-wraparound / FREEZE vacuum. Indeed, by nature an anti-wraparound VACUUM is more likely to touch blocks that weren’t modified for a long time as it’s targeting tuple being visible for more than 200M transactions (by default). Even though it’s only setting a flag bit to mark the tuple as frozen, if that block wasn’t modified since the last CHECKPOINT, this bit will be amplified to a full page image which is way more data.

With this new feature, it’s now possible to really monitor the WAL generation, which will help to better tune your instances!

New in pg13: WAL monitoring was originally published by Julien Rouhaud at rjuju's home on April 07, 2020.

New in pg13: Monitoring the query planner

2020-04-04T12:06:15+00:00

Depending on your workload, the planning time can represent a significant part of the overal query procesing time. This is especially import in OLTP workload, but OLAP queries with numerous tables being joined and an aggressive configuration on the JOIN order search can also lead to hight planning time.

Planning counters in pg_stat_statements

Previously, pg_stat_statements was only keeping track of the execution part of a query processing: the number of execution, cumulated time, but also minimum, maximum, mean and also the standard deviation. With PostgreSQL 13, you’ll also have those metrics for the planification part!

commit 17e03282241c6ac58a714eb0c3b6a8018cf6167a
Author: Fujii Masao <fujii@postgresql.org>
Date:   Thu Apr 2 11:20:19 2020 +0900

    Allow pg_stat_statements to track planning statistics.

    This commit makes pg_stat_statements support new GUC
    pg_stat_statements.track_planning. If this option is enabled,
    pg_stat_statements tracks the planning statistics of the statements,
    e.g., the number of times the statement was planned, the total time
    spent planning the statement, etc. This feature is useful to check
    the statements that it takes a long time to plan. Previously since
    pg_stat_statements tracked only the execution statistics, we could
    not use that for the purpose.

    The planning and execution statistics are stored at the end of
    each phase separately. So there are not always one-to-one relationship
    between them. For example, if the statement is successfully planned
    but fails in the execution phase, only its planning statistics are stored.
    This may cause the users to be able to see different pg_stat_statements
    results from the previous version. To avoid this,
    pg_stat_statements.track_planning needs to be disabled.

    This commit bumps the version of pg_stat_statements to 1.8
    since it changes the definition of pg_stat_statements function.

    Author: Julien Rouhaud, Pascal Legrand, Thomas Munro, Fujii Masao
    Reviewed-by: Sergei Kornilov, Tomas Vondra, Yoshikazu Imai, Haribabu Kommi, Tom Lane
    Discussion: https://postgr.es/m/CAHGQGwFx_=DO-Gu-MfPW3VQ4qC7TfVdH2zHmvZfrGv6fQ3D-Tw@mail.gmail.com
    Discussion: https://postgr.es/m/CAEepm=0e59Y_6Q_YXYCTHZkqOc6H2pJ54C_Xe=VFu50Aqqp_sA@mail.gmail.com
    Discussion: https://postgr.es/m/DB6PR0301MB21352F6210E3B11934B0DCC790B00@DB6PR0301MB2135.eurprd03.prod.outlook.com

Keep in mind that even simple query can have a surprisingly high planification time. One of the frequent cause was the get_actual_variable_range() function, which is called when the planner wants to know what are the minimum and maximum values of a specific field. This function detects if a suitable index exists, and if there’s one it gets the wanted values. However, when there were a lot of uncommitted values at the end of the index range, it could take a significant amount of time to get a visible value. While this problem has been fixed long ago (see this commit and this other commit for more details), there are still some cases where the planning time is higher than what you’d expect, so having an easy way to monitor the planification metrics is worthwhile.

This feature can also be interesting to know how much you’re using the generic plan feature for instance, and how much of a difference this should make for instance.

Let’s see a simple example, to see the effect of generic plans with prepared statements:

=# PREPARE s1 AS SELECT count(*) FROM pg_class;
PREPARE
=# EXECUTE s1;
 count
-------
   387
(1 row)

[... 5 more times ...]

=# SELECT query, plans, total_plan_time, total_plan_time / plans AS avg_plan,
   calls, total_exec_time, total_exec_time / calls AS avg_exec
   FROM pg_stat_statements
   WHERE query ILIKE '%SELECT count(*) FROM pg_class%';
-[ RECORD 1 ]---+--------------------------------------------
query           | PREPARE s1 AS SELECT count(*) FROM pg_class
plans           | 1
total_plan_time | 2.119496
avg_plan        | 2.119496
calls           | 6
total_exec_time | 3.4918280000000004
avg_exec        | 0.5819713333333334

While the query was executed 6 times, it was actually planned only once (since there’s no parameter, a generic plan is always used). While the execution time is on average slightly more than half a milliscond, a single planning was almost 4 times more expensive. By saving 5 planification, postgres saved up to 10ms.

Planning buffers in EXPLAIN

commit ce77abe63cfc85fb0bc236deb2cc34ae35cb5324
Author: Fujii Masao <fujii@postgresql.org>
Date:   Sat Apr 4 03:13:17 2020 +0900

    Include information on buffer usage during planning phase, in EXPLAIN output, take two.

    When BUFFERS option is enabled, EXPLAIN command includes the information
    on buffer usage during each plan node, in its output. In addition to that,
    this commit makes EXPLAIN command include also the information on
    buffer usage during planning phase, in its output. This feature makes it
    easier to discern the cases where lots of buffer access happen during
    planning.

    This commit revives the original commit ed7a509571 that was reverted by
    commit 19db23bcbd. The original commit had to be reverted because
    it caused the regression test failure on the buildfarm members prion and
    dory. But since commit c0885c4c30 got rid of the caues of the test failure,
    the original commit can be safely introduced again.

    Author: Julien Rouhaud, slightly revised by Fujii Masao
    Reviewed-by: Justin Pryzby
    Discussion: https://postgr.es/m/16109-26a1a88651e90608@postgresql.org

Following the same idea, EXPLAIN will now display the buffer usage if the BUFFERS option is used. If you try that on a fresh new connection, before any catalog cache is populated, you could be surprised on how many buffers would be accessed even for a simple query:

=# EXPLAIN (BUFFERS, ANALYZE, COSTS OFF) SELECT * FROM pg_class;
                                               QUERY PLAN
---------------------------------------------------------------------------------------------------------
 Seq Scan on pg_class (actual time=0.028..0.410 rows=388 loops=1)
   Buffers: shared hit=13
 Planning Time: 5.157 ms
   Buffers: shared hit=118
 Execution Time: 1.257 ms
(5 rows)

=# EXPLAIN (BUFFERS, ANALYZE, COSTS OFF) SELECT * FROM pg_class;
                            QUERY PLAN
------------------------------------------------------------------
 Seq Scan on pg_class (actual time=0.035..0.413 rows=388 loops=1)
   Buffers: shared hit=13
 Planning Time: 0.393 ms
 Execution Time: 0.670 ms

We can see here that populating the cache (relation, columns, datatypes…) access 118 blocks, and that’s probably a significant part of the 5 extra ms we saw in the first EXPLAIN output.

New in pg13: Monitoring the query planner was originally published by Julien Rouhaud at rjuju's home on April 04, 2020.

Planner selectivity estimation error statistics with pg_qualstats 2

2020-02-28T12:37:04+00:00

Selectivity estimation error is one of the main cause of bad query plans. It’s quite straighforward to compute those estimation error using EXPLAIN (ANALYZE), either manually or with the help of explain.depesz.com (or other similar tools), but until now there were now tool available to get this information automatically and globally. Version 2 of pg_qualstats fixes that, thanks a lot to Oleg Bartunov for the original idea!

Note: If you don’t know pg_qualstats extension, you may want to see my last article about it.

The problem

There can be many causes to that issue: outdated statistics, complex predicates, non uniform data… But whatever the reason is, if the optimizer doesn’t have an accurate idea on how much data each predicate will filter, the result is the same: a bad query plan, which can lead to longer query execution.

To illustrate the problem, I’ll use here a simple test case, voluntarily built to fool the optimizer.

rjuju=# CREATE TABLE pgqs AS
             SELECT  i%2 val1 , (i+1)%2 val2
             FROM generate_series(1, 50000) i;
SELECT 50000

rjuju=# VACUUM ANALYZE pgqs;
VACUUM

rjuju=# EXPLAIN (ANALYZE) SELECT * FROM pgqs WHERE val1 = 1 AND val2 = 1;
                             QUERY PLAN
--------------------------------------------------------------------
 Seq Scan on pgqs  ([...] rows=12500 width=8) ([...] rows=0 loops=1)
   Filter: ((val1 = 1) AND (val2 = 1))
   Rows Removed by Filter: 50000
 Planning Time: 0.553 ms
 Execution Time: 38.062 ms
(5 rows)

Here postgres think that the query will emit 12500 tuples, while in reality none will be emitted. If you’re wondering how postgres came up with that number, the explanation is simple. When multiple independant (overlapping range predicate can be merged) clauses are AND-ed and no extended statistics are available (see below for more about it), postgres will simply multiply each clause selectivity. This is done in clauselist_selectivity_simple, in src/backend/optimizer/path/clausesel.c:

Selectivity
clauselist_selectivity_simple(PlannerInfo *root,
                List *clauses,
                int varRelid,
                JoinType jointype,
                SpecialJoinInfo *sjinfo,
                Bitmapset *estimatedclauses)
{
  Selectivity s1 = 1.0;
  [...]
  /*
   * Anything that doesn't look like a potential rangequery clause gets
   * multiplied into s1 and forgotten. Anything that does gets inserted into
   * an rqlist entry.
   */
  listidx = -1;
  foreach(l, clauses)
  {
    [...]
    /* Always compute the selectivity using clause_selectivity */
    s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo);
    [...]
        /*
         * If it's not a "<"/"<="/">"/">=" operator, just merge the
         * selectivity in generically.  But if it's the right oprrest,
         * add the clause to rqlist for later processing.
         */
        switch (get_oprrest(expr->opno))
        {
          [...]
          default:
            /* Just merge the selectivity in generically */
            s1 = s1 * s2;
            break;
          [...]

In this case, each predicate will independantly filter approximately 50% of the table, as we can see in pg_stats view:

rjuju=# SELECT tablename, attname, most_common_vals, most_common_freqs
        FROM pg_stats WHERE tablename = 'pgqs';
 tablename | attname | most_common_vals |    most_common_freqs
-----------+---------+------------------+-------------------------
 pgqs      | val1    | {0,1}            | {0.50116664,0.49883333}
 pgqs      | val2    | {1,0}            | {0.50116664,0.49883333}
(2 rows)

So when using both clauses, the estimate is 25% of the table, since postgres doesn’t know by default that both values are mutually exclusive. Continuing with this artificial test case, let’s see what happens if we add a join on top of if. For instance, joining the table to itself on the val1 column only. For clarity, I’ll use t1 for the table on which I’m applying the mutually exclusive predicates, and t2 the table joined:

rjuju=# EXPLAIN ANALYZE SELECT *
        FROM pgqs t1
        JOIN pgqs t2 ON t1.val1 = t2.val1
        WHERE t1.val1 = 0 AND t1.val2 = 0;
                                     QUERY PLAN
-----------------------------------------------------------------------------------
 Nested Loop  ([...] rows=313475000 width=16) ([...] rows=0 loops=1)
   ->  Seq Scan on pgqs t2  ([...] rows=25078 width=8) ([...] rows=25000 loops=1)
         Filter: (val1 = 0)
         Rows Removed by Filter: 25000
   ->  Materialize  ([...] rows=12500 width=8) ([...] rows=0 loops=25000)
         ->  Seq Scan on pgqs t1  ([...] rows=12500 width=8) ([...] rows=0 loops=1)
               Filter: ((val1 = 0) AND (val2 = 0))
               Rows Removed by Filter: 50000
 Planning Time: 0.943 ms
 Execution Time: 86.757 ms
(14 rows)

Postgres thinks that this join will emit 313 millions rows, while obviously no rows will be emitted. And this is a good example on how bad assumptions can lead to an inefficient plan.

Here Postgres can deduce that the val1 = 0 predicate can be applied to t2. So how to join two relations, one that should emit 25000 tuples and the other that should emit 12500 tuples, with no index available? A nested loop is not a bad choice, as both relation aren’t really big. As no index is available, postgres also chooses to materialize the inner relation, meaning storing it in memory, to make it more efficient. As it tries to limit memory consumption as much as possible, the smallest relation is materialized, and that’s the mistake here.

Indeed, postgres will read the whole table twice: once to get every rows corresponding to the val1 = 0 predicate for the outer relation, and once to find all rows to be materialized. If the opposite was done, as it would probably have if the estimates had been more realistic, the table would only have been read once.

In this case, as the dataset isn’t big and quite artificial, a better plan wouldn’t drastically change the execution time. But keep in mind than with real production environements, it could mean choosing a nested loop assuming that there’ll be only a couple of rows to loop on while in reality the backend will spend minutes or even hours looping over millions of rows, and another plan would have been orders of magnitude quicker.

Detecting the problem

pg_qualstats 2 will now compute the selectivity estimation error, both in a ratio and a raw number, and will keep track for each predicate the minimum, maximum and mean values, with the standard deviation. This is now quite simple to detect problematic quals!

After executing the last query, here’s what the pg_qualstats view will return:

rjuju=# SELECT relname, attname, opno::regoper, qualid, qualnodeid,
    mean_err_estimate_ratio mean_ratio, mean_err_estimate_num mean_num, constvalue
    FROM pg_qualstats pgqs
    JOIN pg_class c ON pgqs.lrelid = c.oid
    JOIN pg_attribute a ON a.attrelid = c.oid AND a.attnum = pgqs.lattnum;
 relname | attname | opno |   qualid   | qualnodeid | mean_ratio | mean_num | constvalue
---------+---------+------+------------+------------+------------+----------+------------
 pgqs    | val1    | =    |     <NULL> | 3161070364 | 1.00393542 |       98 | 0::integer
 pgqs    | val1    | =    | 3864967567 | 3161070364 |      12500 |    12500 | 0::integer
 pgqs    | val2    | =    | 3864967567 | 3065200358 |      12500 |    12500 | 0::integer
(3 rows)

NOTE: qualid is an identifier if multiple qual are AND-ed, NULL otherwise, and qualnodeid is a per-qual only identifier.

We see here that when used alone, the qual pgqs.val = ? doesn’t show any selectivity estimate problem as the ratio (mean_ratio) is very close to 1 and the raw number (mean_num) is quite low. On the other hand, when combined with AND pgqs.val2 = ? pg_qualstats reports significant estimate error. That’s a very strong sign that those columns are functionally dependent.

If for example a qual alone shows issues, it could be a sign of outdated statistics, or that the sample size isn’t big enough.

Also, if you have pg_stat_statements extension installed, pg_qualstats will give you the query identifier for each predicate. With that and a bit of SQL, you can for instance find the query with a long average execution time which contains quals for which the selectivity estimation is off by 10 or more.

Interlude: Extended statistics

If you’re wondering how to solve the issue I just explained, the solution is very easy since extended statistics were introduced in PostgreSQL 10, and assuming that you know that’s the root issue. Create an extended statistcs on the related columns, perform an ANALYZE and you’re done!

rjuju=# CREATE STATISTICS pgqs_stats ON val1, val2 FROM pgqs;
CREATE STATISTICS

rjuju=# ANALYZE pgqs;
ANALYZE

rjuju]=# EXPLAIN ANALYZE SELECT *
        FROM pgqs t1
        JOIN pgqs t2 ON t1.val1 = t2.val1
        WHERE t1.val1 = 0 AND t1.val2 = 0 order by t1.val2;
                             QUERY PLAN
-------------------------------------------------------------------------
 Nested Loop  ([...] rows=25002 width=16) ([...] rows=0 loops=1)
   ->  Seq Scan on pgqs t1  ([...] rows=1 width=8) ([...] rows=0 loops=1)
         Filter: ((val1 = 0) AND (val2 = 0))
         Rows Removed by Filter: 50000
   ->  Seq Scan on pgqs t2  ([...] rows=25002 width=8) (never executed)
         Filter: (val1 = 0)
 Planning Time: 0.559 ms
 Execution Time: 39.471 ms
(8 rows)

If you want more details on extended statistics, I recommend looking at the slides from Tomas Vondra’s excellent talk on this subject.

Going further

Tracking the quals in every single qual executed is of course quite expensive, and would significantly impact the performance for any non datawarehouse workload. That’s why pg_qualstats has an option, pg_qualstats.sample_rate, to sample the query that will be processed. This setting is by default set to 1 / max_connections, which will make the overhead quite negligible, but don’t be surprised if you don’t see any qual reported after running a few queries!

But if you’re instead only interested by the quals that has bad selectivity estimation, for instance to detect this class of problem rather than missing indexes, there are two new options available for that:

pg_qualstats.min_err_estimate_ratio
pg_qualstats.min_err_estimate_num

Those options are cumulative and can be changed at anytime, and will limit the quals that pg_qualstats will store to the ones that have a selectivity estimate ratio and/or raw number higher that what you ask. Although those options will help to reduce the performance overhead, they of course can be combined with pg_qualstats.sample_rate if needed.

Conclusion

After introducing the new global index advisor, this article presented a class of problems that are frequently seen as a DBA, and how to detect and solve them.

I believe that those two new features in pg_qualstats will greatly help PostgreSQL databases administration. Also, external tools that aims to solve related issue, such as pg_plan_advsr or AQO could also benefit from pg_qualstats, as they could directly get the exact data they need to be able perform analysis and optimize the queries!

Planner selectivity estimation error statistics with pg_qualstats 2 was originally published by Julien Rouhaud at rjuju's home on February 28, 2020.

New in pg13: New leader_pid column in pg_stat_activity

2020-02-06T12:59:53+00:00

New leader_pid column in pg_stat_activity view

Surprisingly, since parallel query was introduced in PostgreSQL 9.6, it was impossible to know wich backend a parallel worker was related to. So, as Guillaume pointed out, it makes it quite difficult to build simple tools that can sample the wait events related to all process involved in a query. A simple solution to that problem is to export the lock group leader information available in the backend at the SQL level:

commit b025f32e0b5d7668daec9bfa957edf3599f4baa8
Author: Michael Paquier <michael@paquier.xyz>
Date:   Thu Feb 6 09:18:06 2020 +0900

Add leader_pid to pg_stat_activity

This new field tracks the PID of the group leader used with parallel
query.  For parallel workers and the leader, the value is set to the
PID of the group leader.  So, for the group leader, the value is the
same as its own PID.  Note that this reflects what PGPROC stores in
shared memory, so as leader_pid is NULL if a backend has never been
involved in parallel query.  If the backend is using parallel query or
has used it at least once, the value is set until the backend exits.

Author: Julien Rouhaud
Reviewed-by: Sergei Kornilov, Guillaume Lelarge, Michael Paquier, Tomas
Vondra
Discussion: https://postgr.es/m/CAOBaU_Yy5bt0vTPZ2_LUM6cUcGeqmYNoJ8-Rgto+c2+w3defYA@mail.gmail.com

With this change, you can now easily find all processes involved in a parallel query. For instance:

=# SELECT query, leader_pid,
  array_agg(pid) filter(WHERE leader_pid != pid) AS members
FROM pg_stat_activity
WHERE leader_pid IS NOT NULL
GROUP BY query, leader_pid;
       query       | leader_pid |    members
-------------------+------------+---------------
 select * from t1; |      31630 | {32269,32268}
(1 row)

Be careful, as mentionned in the commit message, if the leader_pid is the same as pid, it doesn’t necessarily mean that the backend is currently performing a parallel query, as once set this field is never reset. Also, to avoid extra ovherhead, no additional lock is held while outputting the data. It means that each row is processed independently. So, while quite unlikely, you can get in some circumstances inconsistent data, such as a parallel worker pointing to a pid that already disconnected.

New in pg13: New leader_pid column in pg_stat_activity was originally published by Julien Rouhaud at rjuju's home on February 06, 2020.

pg qualstats 2: Global index advisor

2020-01-06T12:23:29+00:00

Coming up with good index suggestion can be a complex task. It requires knowledge of both application queries and database specificities. Over the year multiple projects tried to solve this problem, one of which being PoWA with the version 3, with the help of pg_qualstats extension. It can give pretty good index suggestion, but it requires to install and configure PoWA, while some users wanted to only have the global index advisor. In such case and for simplicity, the algorithm used in PoWA is now available in pg_qualstats version 2 without requiring any additional component.

EDIT: The pg_qualstats_index_advisor() function has been changed to return json rather than jsonb, so that the compatibility with PostgreSQL 9.3 is maintained. The query examples are therefore also modified to use json_array_elements() rather than jsonb_array_elements().

What is pg_qualstats

A simple way to explain what is pg_qualstats would be to say that it’s like pg_stat_statements working at the predicate level.

The extension will save useful statistics for WHERE and JOIN clauses: which table and column a predicate refers to, number of time the predicate has been used, number of execution of the underlying operator, whether it’s a predicate from an index scan or not, selectivity, constant values used and much more.

You can deduce many things from such information. For instance, if you examine the predicates that contains references to different tables, you can find which tables are joined together, and how selective are those join conditions.

Global suggestion?

As I mentioned, the global index advisor added in pg_qualstats 2 uses the same approach as the one in PoWA, so the explanation here will describe both tools. The only difference is that with PoWA you’ll likely get a better suggestion, as more predicates will be available, and you can also choose for wich time interval you want to detect missing indexes.

The important thing here is that the suggestion is performed globally, considering all interesting predicates at the same time. This approach is different to all other approaches I saw that only consider a single query at a time. I believe that a global approach is better, as it’s possible to reduce the total number of indexes, maximizing multi-column indexes usefulness.

How global suggestion is done

The first step is to gather all predicates that could benefit from a new index. This is easy to get with pg_qualstats, by filtering the predicates coming from sequential scans, executed many time, that filter many rows (both in number of rows and in percentage) you get a perfect list of predicates that likely miss an index (or alternatively the list of poorly written queries in certain cases). For instance, let’s consider an application which uses those 4 predicates:

Next, we build the full set of paths with each AND-ed predicates that contains other, also possibly AND-ed, predicates. Using the same 4 predicates, we would get those paths:

Once all the paths are built, we just need to get the best path to find out the best index to suggest. The scoring is for now done by giving a weight to each node of each path corresponding to the number of simple predicates it contains and summing the weight for each path. This is very simple and allows to prefer a smaller amount of indexes to optimize as many queries as possible. With our simple example, we get:

Of course, other scoring approaches could be used to take into account other parameters and give possibly better suggestions. For instance, combining the number of executions or the predicate selectivity. If the read/write ratio for each table is known (this is available using powa-archivist), it would also be possible to adapt the scoring method to limit index suggestions for write-mostly tables. With this algorithm, all of that could be added quite easily.

Once the best path is found, we can generate an index DDL! As the order of the columns can be important, this is done using getting the columns for each node in ascending weight order. In our example, we would generate this index:

CREATE INDEX ON t1 (id, ts, val);

Once an index is found, we simply remove the contained predicates for the global list of predicates and start again from scratch until there are no predicate left.

Additional details and caveat

Of course, this is a simplified version of the suggestion algorithm. Some other informations are required. For instance, the list of predicates is actually expanded with operator classes and access method depending on the column types and operator, to make sure that the suggested indexes are valid. If multiple index methods are found for a best path, btree will be chosen in priority.

This brings another consideration: this approach is mostly thought for btree indexes, for which the column order is critical. Some other access methods don’t require a specific column order, and for those it could be possible to get better index suggestions if the column order parameters wasn’t considered.

Another important point is that the operator classes and access method is not hardcoded but retrieved at execution time using the local catalogs. Therefore, you can get different (and possibly better) results if you make sure that optional operator classes are present when using the index advisor. This could be btree_gist or btree_gin extensions, but also other access methods. It’s also possible that some type / operator combination doesn’t have any associated access method recorded in the catalogs. In this case, those predicates are returned separately as a list of unoptimizable predicates, that should be manually analyzed.

Finally, as pg_qualstats isn’t considering expression predicates, this advisor can’t suggest indexes on expression, for instance if you’re using fulltext search.

Usage example

A simple set-returning function is provided, with optional parameters, that returns a json value:

CREATE OR REPLACE FUNCTION pg_qualstats_index_advisor (
    min_filter integer DEFAULT 1000,
    min_selectivity integer DEFAULT 30,
    forbidden_am text[] DEFAULT '{}')
    RETURNS json

The parameter names are self explanatory:

min_filter: how many tuples should a predicate filter on average to be considered for the global optimization, by default 1000.
min_selectivity: how selective should a predicate filter on average to be considered for the global optimization, by default 30%.
forbidden_am: list of access methods to ignore. None by default, although for PostgreSQL 9.6 and prior hash indexes will internally be discarded, as those are only safe since version 10.

Using pg_qualstats regression tests, let’s see a simple example:

CREATE TABLE pgqs AS SELECT id, 'a' val FROM generate_series(1, 100) id;
CREATE TABLE adv (id1 integer, id2 integer, id3 integer, val text);
INSERT INTO adv SELECT i, i, i, 'line ' || i from generate_series(1, 1000) i;
SELECT pg_qualstats_reset();
SELECT * FROM adv WHERE id1 < 0;
SELECT count(*) FROM adv WHERE id1 < 500;
SELECT * FROM adv WHERE val = 'meh';
SELECT * FROM adv WHERE id1 = 0 and val = 'meh';
SELECT * FROM adv WHERE id1 = 1 and val = 'meh';
SELECT * FROM adv WHERE id1 = 1 and id2 = 2 AND val = 'meh';
SELECT * FROM adv WHERE id1 = 6 and id2 = 6 AND id3 = 6 AND val = 'meh';
SELECT * FROM adv WHERE val ILIKE 'moh';
SELECT COUNT(*) FROM pgqs WHERE id = 1;

And here’s what the function returns:

SELECT v
  FROM json_array_elements(
    pg_qualstats_index_advisor(min_filter => 50)->'indexes') v
  ORDER BY v::text COLLATE "C";
                               v
---------------------------------------------------------------
 "CREATE INDEX ON public.adv USING btree (id1)"
 "CREATE INDEX ON public.adv USING btree (val, id1, id2, id3)"
 "CREATE INDEX ON public.pgqs USING btree (id)"
(3 rows)

SELECT v
  FROM json_array_elements(
    pg_qualstats_index_advisor(min_filter => 50)->'unoptimised') v
  ORDER BY v::text COLLATE "C";
        v
-----------------
 "adv.val ~~* ?"
(1 row)

The version 2 of pg_qualstats is not released yet, but feel free to test it and report any issue you may find!

pg qualstats 2: Global index advisor was originally published by Julien Rouhaud at rjuju's home on January 06, 2020.

PoWA 4: New powa-collector daemon

2019-12-10T18:54:17+00:00

This article is part of the PoWA 4 beta series, and describes the new powa-collector daemon.

New powa-collector daemon

This daemon replaces the previous background worker when using the new remote mode. It’s a simple daemon written in python, which will perform all the required steps to perform remote snapshots. It’s available on pypi.

As I explained in my previous article introducing PoWA 4, this daemon is required for a remote mode setup, with this architecture in mind:

Its configuration is very simple. All you need to do is copy and rename the provided powa-collector.conf.sample file, and adapt the connection URI to describe how to connect on your dedicated repository server, and you’re done.

A typical configuration will look like:

{
    "repository": {
        "dsn": "postgresql://powa_user@server_dns:5432/powa",
    },
    "debug": true
}

The list of remote servers, their configuration and everything else it needs will be automatically retrieved from the repository server you just configured. When started, it’ll spawn one dedicated thread per declared remote server, and maintain a persistent connection on the configured powa database on this remote server. Each thread will perform a remote snapshot, exporting the data on the repository server using the new source functions. Each thread will open and close a connection on the repository server when performing the remote snapshot.

This daemon obviously needs to be able to connect to all the declared remote servers and the repository server. The powa_servers table, which store the list of remote servers, has a field to store username and password to connect to the remote server. Storing a password in plain text in this table is an heresy as far as security is concerned. So, as mentioned in the PoWA security documentation, you can store a NULL password and instead use any of the authentication method that libpq supports (.pgpass file, certificate…). That’s strongly recommended for any non toy setup.

The persistent connection on the repository server is used to monitor the daemon:

to check that the daemon is up and running
to communicate through the UI using a simple protocol to perform various actions (reload the configuration, check for a remote server thread status…)

Note that you can also ask the daemon to reload its configuration by issuing a SIGHUP to the daemon process. A reload is required if any modification to the list of remote servers (if you added or removed a remote server, or updated a setting for an existing) has been done.

Also note that by choice, powa-collector will not perform local snapshots. If you want to use PoWA for the repository server, you need to enable the original background worker.

New configuration page

The configuration page is now updated to give all needed information about the background worker status and the powa-collector daemon status (including all of its dedicated threads) and the list of registered remote servers. Here’s an example of the new root configuration page:

If the powa-collector daemon is used, each remote server status will be retrieved using the communication protocol. If the collector encountered any error (connecting to a remote server, during a snapshot or anything else), they’ll also be displayed here. Also note that such errors will also be displayed on top of any page of the UI, so that you can’t miss them.

Also, the configuration section has now a hierarchy, and you’ll be able to see the list of extensions and the current PostgreSQL configuration for the local or remote servers by clicking on the server of your choice!

There’s also a new Reload collector button on the header panel, which as expected will ask the collector to reload its configuration. That can be useful if you registered new servers and you don’t have access on the server where the collector is running.

Conclusion

This is the last article introducing the new version of PoWA. It’s still in beta, so feel free to test it, report any issue you may find or give any other feedback!

PoWA 4: New powa-collector daemon was originally published by Julien Rouhaud at rjuju's home on December 10, 2019.

PoWA 4: changes in powa-archivist!

2019-06-05T14:26:17+00:00

This article is part of the PoWA 4 beta series, and describes the changes done in powa-archivist.

For more information about this v4, you can consult the general introduction article.

Quick overview

First of all, you have to know that there is not upgrade possible from v3 to v4, so a DROP EXTENSION powa is required if you were already using PoWA on any of your servers. This is because this v4 involved a lot of changes in the SQL part of the extension, making it the most significant change in the PoWA suite for this new version. Looking at the amount changes at the time I’m writing this article, I get:

 CHANGELOG.md       |   14 +
 powa--4.0.0dev.sql | 2075 +++++++++++++++++++++-------
 powa.c             |   44 +-
 3 files changed, 1629 insertions(+), 504 deletions(-)

The lack of upgrade shouldn’t be a problem in practice though. PoWA is a performance tool, so it’s intended to have data with high precision but with a very limited history. If you’re looking for a general monitoring solution keeping months of counters, PoWA is definitely not the tool you need.

Configuring the list of remote servers

Concerning the features themselves, the first small change is that powa-archivist does not require the background worker to be active anymore, as it won’t be used for remote setup. That means that a PostgreSQL restart is not needed needed anymore to install PoWA. Obviously, a restart is still required if you want to use the local setup, using the background worker, or if you want to install additional extensions that themselves require a restart.

Then, as PoWA needs some configuration (frequency of snapshot, data retention and so on), some new tables are added to be able to configure all of that. The new powa_servers table stores the configuration for all the remote instances whose data should be stored on this instance. This local PoWA instance is call a repository server (that typically should be dedicated to storing PoWA data), in opposition to remote instances which are the instances you want to monitor. The content of this table is pretty straightforward:

\d powa_servers
                              Table "public.powa_servers"
  Column   |   Type   | Collation | Nullable |                 Default
-----------+----------+-----------+----------+------------------------------------------
 id            | integer  |           | not null | nextval('powa_servers_id_seq'::regclass)
 hostname      | text     |           | not null |
 alias         | text     |           |          |
 port          | integer  |           | not null |
 username      | text     |           | not null |
 password      | text     |           |          |
 dbname        | text     |           | not null |
 frequency     | integer  |           | not null | 300
 powa_coalesce | integer  |           | not null | 100
 retention     | interval |           | not null | '1 day'::interval

If you already used PoWA, you should recognize most of the configuration options, that are now stored here. The new options are used to describe how to connect to the remote servers, and can provide an alias to be displayed in the UI.

You also probably noticed a password column here. Storing a password in plain text in this table is an heresy as far as security is concerned. So, as mentioned in the PoWA security section of the documentation, you can store a NULL password and use instead any of the authentication method that libpq supports (.pgpass file, certificate…). That’s strongly recommended for any non toy setup.

Another table, the powa_snapshot_metas table, is also added to store some metadata regarding each remote server snapshot information:

                                   Table "public.powa_snapshot_metas"
    Column    |           Type           | Collation | Nullable |                Default
--------------+--------------------------+-----------+----------+---------------------------------------
 srvid        | integer                  |           | not null |
 coalesce_seq | bigint                   |           | not null | 1
 snapts       | timestamp with time zone |           | not null | '-infinity'::timestamp with time zone
 aggts        | timestamp with time zone |           | not null | '-infinity'::timestamp with time zone
 purgets      | timestamp with time zone |           | not null | '-infinity'::timestamp with time zone
 errors       | text[]

That’s basically a counter to track the number of snapshots done, the timestamp for each kind of event that happened (snapshot, aggregate and purge), and a text array to store any error happening during the snapshot, that the UI can display.

SQL API to configure the remote servers

While thoses table are simple, a basic SQL API is available to register new servers and configure them. Basically, 6 functions are available:

powa_register_server(), to declare a new remote server, and the list of extensions available on it
powa_configure_server() to update any setting for the specified remote server (using a JSON where the key is the name of the parameter to change, and the value is the new value to use)
powa_deactivate_server() to disable snapshots on the specified remote server (which actually is setting up the frequency to -1)
powa_delete_and_purge_server() to remove the specified remote server from the list of servers and remove all associated snapshot data
powa_activate_extension(), to declare that a new extension is available on the specified remote server
powa_deactivate_extension(), to specify that an extension is not available anymore on the specified remote server

Any action more complicated than this should be performed using plain SQL queries. Hopefully, there shouldn’t be many other needs, and the tables are straightforward so this shouldn’t be a problem. Feel free to ask for more functions if you feel the need though. Please also note that the UI doesn’t allow you to call those functions, as the UI is for now entirely read only.

Performing remote snapshots

As metrics are now stored on a different PostgreSQL instance, we had to extensively change the way snapshots (retrieving the data from a stat extension and storing them in PoWA catalog in a space efficient way) are performed.

The list of all stat extensions, or data sources, that are available on a server (either remote or local) and for which we should perform a snapshot are configured in a table called powa_functions:

               Table "public.powa_functions"
     Column     |  Type   | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
 srvid          | integer |           | not null |
 module         | text    |           | not null |
 operation      | text    |           | not null |
 function_name  | text    |           | not null |
 query_source   | text    |           |          |
 added_manually | boolean |           | not null | true
 enabled        | boolean |           | not null | true
 priority       | numeric |           | not null | 10

A new query_source field is added, that provides the name of a source function, required to support remote snapshot of any stat extensions. This function is used to export the counters provided by this extension on a different server, in a dedicated transient table. The snapshot function will then perform the snapshot using those exported data instead of the one provided by stat extensions locally when the remote mode is used. Note that the counters export and the remote snapshot is done automatically with the the new powa-collector daemon, that I’ll cover in another article.

Here’s an example of how PoWA perform a remote snapshot of the list of databases. As you’ll see, this is very simplistic, meaning that it’s very easy to add support for a new stat extension.

The transient table:

   Unlogged table "public.powa_databases_src_tmp"
 Column  |  Type   | Collation | Nullable | Default
---------+---------+-----------+----------+---------
 srvid   | integer |           | not null |
 oid     | oid     |           | not null |
 datname | name    |           | not null |

For better performance, all the transient tables are unlogged, as their content is only needed during a snapshot and are trashed afterwards. In this example the transient table only stores the server identifier for which the data are, the oid and name of each databases present on the remote server.

And the source function:

CREATE OR REPLACE FUNCTION public.powa_databases_src(_srvid integer,
    OUT oid oid, OUT datname name)
 RETURNS SETOF record
 LANGUAGE plpgsql
AS $function$
BEGIN
    IF (_srvid = 0) THEN
        RETURN QUERY SELECT d.oid, d.datname
        FROM pg_database d;
    ELSE
        RETURN QUERY SELECT d.oid, d.datname
        FROM powa_databases_src_tmp d
        WHERE srvid = _srvid;
    END IF;
END;
$function$

This function simply returns the content of pg_database if local data are asked (server id 0 is always the local server), or the content of the transient table for the given remote server otherwise.

The snapshot function can then easily do any required work with the data for the wanted remote server. In the case of the powa_databases_snapshot() function, the just synchronizing the list of databases, and storing the timestamp of removal if a previously existing database is not found anymore.

For more details, you can consult the PoWA datasource integration documentation, which was updated for the version 4 specificities.

PoWA 4: changes in powa-archivist! was originally published by Julien Rouhaud at rjuju's home on June 05, 2019.

PoWA 4 brings a remote mode, available in beta!

2019-05-17T11:04:17+00:00

PoWA 4 is available in beta.

New remote mode!

The new remote mode is the biggest feature introduced in PoWA 4, though there have been other improvements.

I’ll describe here what this new mode implies and what changed in the UI.

If you’re interested in more details about the rest of the changes in PoWA 4, I’ll soon publish other articles for that.

For the most hurried people, feel free to directly go on the v4 demo of PoWA, kindly hosted by Adrien Nayrat. No credential needed, just click on “Login”.

Why is a remote mode important

This feature has probably been the most frequently asked since PoWA was first released, back in 2014. And that was asked for good reasons, as a local mode have some drawbacks.

First, let’s see how was the architecture up to PoWA 3. Assuming an instance with 2 databases (db1 and db2), plus one database dedicated for PoWA. This dedicated database contains both the stat extension required to get the live performance data and to store them.

A custom background worker is started by PoWA, which is responsible for taking snapshots and storing them in the dediacted powa database regularly. Then, using powa-web, you can see the activity of any of the local databases querying the stored data on the dedicated database, and possibly connect to one of the other local database when complete data are needed, for instance when using the index suggestion tool.

With version 4, the architecture with a remote setup change quite a lot:

You can see the a dedicated powa database is still required, but only for the stat extensions. Data are now stored on a different instance. Then, the background worker is replaced by a new collector daemon, which reads the performance data from the remote servers, and store them on the dedicated repository server. Powa-web will then be able to display the activity connecting on the repository server, and also on the remote server when complete data are needed.

In short, with the new remote mode introduced in this version 4:

a PostgreSQL restart is not required anymore to install powa-archivist extension, as the background worker is not mandatory anymore
there is no overhead due to storing and querying data on the same PostgreSQL server as your production server (there are still some part of the UI that requires querying the original server, for instance when showing EXPLAIN plans, but that’s a negligible overhead)
it’s now possible to use PoWA on a hot-standby server

The UI will therefore now welcome you with a initial page to let you chose which server stored on the configured database you want to wotk on:

The main reason it took so much time to bring a remote mode is because this adds quite some complexity, requiring a major rewrite of the whole PoWA stack. We also wanted to add more feature first, such as the global index suggestion, with validation using hypopg introduced with PoWA 3.

Changes in powa-web

The user interface is the component which probably has the most visible changes in this version 4. Here are the most important ones.

Remote mode compatibility

The biggest change is obviously the support for the new remote mode. As a consequence, the first page shown is now a server selector page, displaying all registered remote servers. After choosing the wanted remote server (or local server if you don’t use the remote mode), all other pages will be similar to the one that were available until PoWA 3, but displaying data for a specific remote server only, and of course retrieving the data from the repository powa database, and with some new information I’ll describe just after.

Note that as the data is now stored on a dedicated repository server when using the remote mode, most of the UI is usable without connecting on the currently selected remote server. However, powa-web still requires to connect on the remote server when the original data are needed (for instance, for index suggestion or when showing EXPLAIN plans). The same authentication considerations and possibilities as for the new powa-collector daemon (which will be described in a following article) applies here.

pg_track_settings support

When this extension is properly configured, a new timeline widget will appear, placed between each graph and its overview, displaying any kind of recorded change if any was detected in the currently selected time interval. On the per-database and per-query pages, this list will be filtered by the selected database.

The same timeline will be displayed on every graph of each page, to easily check if this change had any visible impact using the various graphs.

Note that details of the changes will be displayed on mouseover. You can also click on any event on the timeline to make the event stay displayed, and draw a vertical line on the underlying graph.

Here’s an example of such detected configuration change in action:

Please also note that you need at least version 2.0.0 of pg_track_settings, and that the extension has to be installed both on the remote servers and the repository server.

New graphs available

When pg_stat_kcache is setup, its information were previously only displayed on the per-query page. They’re now displayed on per-server and per-database too, in two graphs:

in the Block Access graph, where the OS cache and disk read metrics will replace the read metric
in a new System Resources graph (which is also added in the per-query page), showing the metrics added in pg_stat_kcache 2.1

Here is an example of this new System Resources graph:

There was also a Wait Events graph (available when pg_wait_sampling extension is setup) only available on the per-query page. This graph is now available on the per-server and per-database pages too.

Metrics documentation and documentation link

Some metrics displayed in the user interface was quite self explanatory, while some could be a little bit obscure. Unfortunately, until now there wasn’t any documentation for any of the metrics. That’s now fixed, and all graphs have an information icon, that will display a description of the metrics used in the graph on mouseover. Some graphs will also include a link to the underlying stat extension in PoWA documentation for users who want to learn more about them.

Here’s an example:

And general bugfixes

Some longstanding issues were also reported:

the graph hover box showing metric values had a wrong vertical position
the time selection using the graph preview didn’t show a correct preview after applying the selection
errors on hypothetical index creation or in certain cases their display wasn’t correctly handled in multiple pages
grid filters weren’t reapplied when time selection was changed

If you have ever been annoyed by any of this, you’ll be glad to know that they’re now all fixed!

Conclusion

This 4th version of PoWA represents a lot of time on development, documentation improvements and testing. We’re now quite satisfied with it, but we may have missed some bugs. If you’re interested in this project, I hope that you’ll consider testing the beta, and if needed don’t hesitate to report a bug!

PoWA 4 brings a remote mode, available in beta! was originally published by Julien Rouhaud at rjuju's home on May 17, 2019.

New in pg12: Statistics on checkums errors

2019-04-18T11:02:26+00:00

Data checksums

Added in PostgreSQL 9.3, data checksums can help to detect data corruption happening on the storage side.

Checksums are only enabled if the instance was setup using initdb --data-checksums (which isn’t the default behavior), or if activated afterwards with the new pg_checksums tool also added in PostgreSQL 12.

When enabled, checksums are written each time a block is written to disk, and verified each time a block is read from disk (or from the operating system cache). If the checksum verification fails, an error is reported in the logs. If the block was read by a backend, the query will obviously fails, but if the block was read by a BASE_BACKUP operation (such as pg_basebackup), the command will continue its processing . While data checkums will only catch a subset of possible problems, they still have some values, especially if you don’t trust your storage reliability.

Up to PostgreSQL 11, any checksum validation error could only be found by looking into the logs, which clearly isn’t convenient if you want to monitor such error.

New counters available in pg_stat_database

To make checksum errors easier to monitor, and help users to react as soon as such a problem occurs, PostgreSQL 12 adds new counters in the pg_stat_database view:

commit 6b9e875f7286d8535bff7955e5aa3602e188e436
Author: Magnus Hagander <magnus@hagander.net>
Date:   Sat Mar 9 10:45:17 2019 -0800

Track block level checksum failures in pg_stat_database

This adds a column that counts how many checksum failures have occurred
on files belonging to a specific database. Both checksum failures
during normal backend processing and those created when a base backup
detects a checksum failure are counted.

Author: Magnus Hagander
Reviewed by: Julien Rouhaud

commit 77bd49adba4711b4497e7e39a5ec3a9812cbd52a
Author: Magnus Hagander <magnus@hagander.net>
Date:   Fri Apr 12 14:04:50 2019 +0200

    Show shared object statistics in pg_stat_database

    This adds a row to the pg_stat_database view with datoid 0 and datname
    NULL for those objects that are not in a database. This was added
    particularly for checksums, but we were already tracking more satistics
    for these objects, just not returning it.

    Also add a checksum_last_failure column that holds the timestamptz of
    the last checksum failure that occurred in a database (or in a
    non-dataabase file), if any.

    Author: Julien Rouhaud <rjuju123@gmail.com>

commit 252b707bc41cc9bf6c55c18d8cb302a6176b7e48
Author: Magnus Hagander <magnus@hagander.net>
Date:   Wed Apr 17 13:51:48 2019 +0200

    Return NULL for checksum failures if checksums are not enabled

    Returning 0 could falsely indicate that there is no problem. NULL
    correctly indicates that there is no information about potential
    problems.

    Also return 0 as numbackends instead of NULL for shared objects (as no
    connection can be made to a shared object only).

    Author: Julien Rouhaud <rjuju123@gmail.com>
    Reviewed-by: Robert Treat <rob@xzilla.net>

Those counters will reflect checksum validation errors for both backend activity and BASE_BACKUP activity, per database.

rjuju=# \d pg_stat_database
                        View "pg_catalog.pg_stat_database"
        Column         |           Type           | Collation | Nullable | Default
-----------------------+--------------------------+-----------+----------+---------
 datid                 | oid                      |           |          |
 datname               | name                     |           |          |
 [...]
 checksum_failures     | bigint                   |           |          |
 checksum_last_failure | timestamp with time zone |           |          |
 [...]
 stats_reset           | timestamp with time zone |           |          |

The checksum_failures column will show a cumulated number of errors, and the checksum_last_failure column will show the timestamp of the last checksum failure on the database (NULL if no error ever happened).

To avoid any confusion (thanks to Robert Treat for pointing it), those two columns will always return NULL if data checksums aren’t enabled, so people won’t mistakenly think that data checksums are always successfully verified.

As a side effect, pg_stat_database will also now show available statistics for shared objects (such as the pg_database table for instance), in a new row with datid valued to 0, and a NULL datname. Those were always accumulated, but weren’t displayed in any system view until now.

~~A dedicated check is also already planned in check_pgactivity!~~ A dedicated check is also already available in check_pgactivity!

New in pg12: Statistics on checkums errors was originally published by Julien Rouhaud at rjuju's home on April 18, 2019.

pg_stat_kcache 2.1 is out

2018-07-17T17:34:13+00:00

A new version of pg_stat_kcache is out, with support for Windows and other platforms, and more counters available.

What’s new

Version 2.1 of pg_stat_kcache has just been released.

The two main new features are:

compatibility with platform without getrusage() support (such as Windows)
more fields of getrusage() are exposed

As I explained in a previous article, this extension is a wrapper on top of getrusage, that accumulates performance counters per normalized query. It was already giving some precious informations that allows a DBA to identify CPU-intensive queries, or compute a real hit-ratio for instance.

However, it was only available on platforms that have a native version getrusage, so Windows and some other platforms were not supported. But fortunately, PostgreSQL does offer a basic support of getrusage() for those platforms. This infrastructure has been used in the version 2.1.0 of pg_stat_kcache, which means that you can now use this extension on Windows and all the other platforms that wasn’t supported previously. As this is a limited support, only the user and system CPU metrics will be available, the other fields will always be NULL.

This new version also exposes all the remaining fields of getrusage() that have a meaning when accumulated per query:

soft page faults
hard page faults
swaps
IPC messages sent and received
signals received
voluntary and involuntary context switches

Another change is to automatically detect the operating system’s clock tick. Otherwise, very short queries (faster than a clock tick) would be either detected as not consuming CPU time, or consuming CPU time from earlier short queries. For queries faster than 3 clock ticks, where imprecision is high, pg_stat_kcache will instead use the query duration as CPU user time, and won’t use anything as CPU system time.

Small example

Depending on your platform, some of those new counters aren’t maintained. On GNU/Linux for instance , the swaps, IPC messages and signaled are unfortunately not maintained, but those which are are still quite interesting. For instance, let’s compare the context switches if we run the same number of total transaction but with either 2 or 80 concurrent connections on a 4-core laptop:

psql -c "SELECT pg_stat_kcache_reset()"
pgbench -c 80 -j 80 -S -n pgbench -t 100
[...]
number of transactions actually processed: 8000/8000
latency average = 8.782 ms
tps = 9109.846256 (including connections establishing)
tps = 9850.666577 (excluding connections establishing)

psql -c "SELECT user_time, system_time, minflts, majflts, nvcsws, nivcsws FROM pg_stat_kcache WHERE datname = 'pgbench'"
     user_time     |    system_time     | minflts | majflts | nvcsws | nivcsws
-------------------+--------------------+---------+---------+--------+---------
 0.431648000000005 | 0.0638690000000001 |   24353 |       0 |     91 |     282
(1 row)

psql -c "SELECT pg_stat_kcache_reset()"
pgbench -c 2 -j 2 -S -n pgbench -t 8000
[...]
number of transactions actually processed: 8000/8000
latency average = 0.198 ms
tps = 10119.638426 (including connections establishing)
tps = 10188.313645 (excluding connections establishing)

psql -c "SELECT user_time, system_time, minflts, majflts, nvcsws, nivcsws FROM pg_stat_kcache WHERE datname = 'pgbench'"
     user_time     | system_time | minflts | majflts | nvcsws | nivcsws 
-------------------+-------------+---------+---------+--------+---------
 0.224338999999999 |    0.023669 |    5983 |       0 |      0 |       8
(1 row)

As expected, having 80 concurrent connections on a 4-core laptop is not the most efficient way to process 8000 transactions. The transactions latency is 44 times slower with 80 connections than with 2 connections. At the O/S level, we can see that with only 2 concurrent connections, we had only 8 involuntary context switches on all queries on the pgbench database, while there were 282, so 35 times more with 80 concurrent connections.

Those new metrics give a lot more information of what’s happening at the O/S level, on a per normalized query granularity, and will ease diagnostic of performance issues. Combined with PoWA, you’ll even be able to identify when any of those metrics have a different behavior!

pg_stat_kcache 2.1 is out was originally published by Julien Rouhaud at rjuju's home on July 17, 2018.

Wait Events support for PoWA

2018-07-09T10:43:34+00:00

You can now view the Wait Events in PoWA thanks to the pg_wait_sampling extension.

Wait Events & pg_wait_sampling

Wait events are a famous and useful feature in a lot of RDBMS. They have been added in PostgreSQL 9.6, quite a few versions ago. Unlike most of others PostgreSQL statistics, those are only an instant view of what the processes are currently waiting on, and not some cumulated counters. You can get those event using the pg_stat_activity view, for instance:

=# SELECT datid, pid, wait_event_type, wait_event, query FROM pg_stat_activity;
 datid  |  pid  | wait_event_type |     wait_event      |                                  query
--------+-------+-----------------+---------------------+-------------------------------------------------------------------------
 <NULL> | 13782 | Activity        | AutoVacuumMain      |
  16384 | 16615 | Lock            | relation            | SELECT * FROM t1;
  16384 | 16621 | Client          | ClientRead          | LOCK TABLE t1;
 847842 | 16763 | LWLock          | WALWriteLock        | END;
 847842 | 16764 | Lock            | transactionid       | UPDATE pgbench_branches SET bbalance = bbalance + 1229 WHERE bid = 1;
 847842 | 16766 | LWLock          | WALWriteLock        | END;
 847842 | 16767 | Lock            | transactionid       | UPDATE pgbench_tellers SET tbalance = tbalance + 3383 WHERE tid = 86;
 847842 | 16769 | Lock            | transactionid       | UPDATE pgbench_branches SET bbalance = bbalance + -3786 WHERE bid = 10;
[...]

In this example, we can see that the wait event for pid 16615 is a Lock on a Relation. In other words, the query is blocked waiting for a heavyweight lock, while the pid 16621, which obviously holds the lock, is idle waiting for client commands. This is something we could already see in previous version, though in a different manner. But more interesting, we can also see that the wait event for pid 16766 is a LWLock, or a Lightweight Lock. Those are internal transient locks that you previsouly couldn’t see at the SQL level. In this example, the query is waiting for the WALWriteLock, a lightweight lock mainly used to control the write to WAL buffers. A complete list of the available wait events is available on the official documentation.

Those information were lacking and are helpful to diagnose bottlenecks. However, only having an instant view of those wait events is certainly not enough to have a good idea of what’s happening on a server. Since most of the wait events are by nature transient events, what you need is to sample them at some high frequency. Trying to sample them with some external tool, even at a second interval, is usually not enough. That’s where the pg_wait_sampling extension bring a really cool solution. It’s an extension written by Alexander Korotkov and Ildus Kurbangaliev. Once activated (it requires to be setup in shared_preload_libraries, so a PostgreSQL restart is necessary), it’ll sample the wait events in shared memory every 10 ms (by default), and also aggregate the counters per wait_event_type, wait_event, pid and queryid (if pg_stat_statements is also activated). For more details on how to configure and use it, you can refer to the extension’s README. Since the work is done in memory as a C extension, it’s very efficient. It’s also implemented with very few locking, so its overhead should be almost negligible. I did some benchmarking on my laptop (I unfortunately don’t have better machine to test on) with a read-only pgbench where all the data fit in shared_memory, with both 8 and 90 clients, to try to get a maximal overhead. The average of 3 runs was around 1%, while the fluctuations between runs was around 0.8%.

And PoWA?

So, thanks to this extension, we now have a cumulated and extremely precise view of the wait events. That’s quite nice, but as the other cumulated statistics available in PostgreSQL, you need to sample the counters regularly if you want to be able to know what happened at a given time in the past, as stated in the README:

[…] Waits profile. It’s implemented as in-memory hash table where count of samples are accumulated per each process and each wait event (and each query with pg_stat_statements). This hash table can be reset by user request. Assuming there is a client who periodically dumps profile and resets it, user can have statistics of intensivity of wait events among time.

That’s exactly the aim of PoWA: save the statistic counters in an efficient way and display them on a GUI.

PoWA 3.2 will automatically detect if the pg_wait_sampling extension is already present or if you install it subsequently and will start to snapshot its data, giving a really precise view of the wait events on your databases over time!

The data is gathered in standard PoWA tables, powa_wait_sampling_history_current for the last 100 (default powa.coalesce) snapshots, and the older values are aggregated in powa_wait_sampling_history, with up to powa.retention history. For instance, here’s a simple query displaying the first 20 changes that occured in the last 100 snapshots:

WITH s AS (
SELECT (record).ts, queryid, event_type, event,
(record).count - lag((record).count)
    OVER (PARTITION BY queryid, event_type, event ORDER BY (record).ts)
    AS events
FROM powa_wait_sampling_history_current w
JOIN pg_database d ON d.oid = w.dbid
WHERE d.datname = 'bench'
)
SELECT *
FROM s
WHERE events != 0
ORDER BY ts ASC, event DESC
LIMIT 20;
              ts               |       queryid        | event_type |     event      | events
-------------------------------+----------------------+------------+----------------+--------
 2018-07-09 10:44:08.037191+02 | -6531859117817823569 | LWLock     | pg_qualstats   |   1233
 2018-07-09 10:44:28.035212+02 |  8851222058009799098 | Lock       | tuple          |      4
 2018-07-09 10:44:28.035212+02 | -6860707137622661878 | Lock       | tuple          |    149
 2018-07-09 10:44:28.035212+02 |  8851222058009799098 | Lock       | transactionid  |    193
 2018-07-09 10:44:28.035212+02 | -6860707137622661878 | Lock       | transactionid  |   1143
 2018-07-09 10:44:28.035212+02 | -6531859117817823569 | LWLock     | pg_qualstats   |      1
 2018-07-09 10:44:28.035212+02 |  8851222058009799098 | LWLock     | lock_manager   |      2
 2018-07-09 10:44:28.035212+02 | -6860707137622661878 | LWLock     | lock_manager   |      3
 2018-07-09 10:44:28.035212+02 | -6860707137622661878 | LWLock     | buffer_content |      2
 2018-07-09 10:44:48.037205+02 |  8851222058009799098 | Lock       | tuple          |     14
 2018-07-09 10:44:48.037205+02 | -6860707137622661878 | Lock       | tuple          |    335
 2018-07-09 10:44:48.037205+02 | -6860707137622661878 | Lock       | transactionid  |   2604
 2018-07-09 10:44:48.037205+02 |  8851222058009799098 | Lock       | transactionid  |    384
 2018-07-09 10:44:48.037205+02 | -6860707137622661878 | LWLock     | lock_manager   |     13
 2018-07-09 10:44:48.037205+02 |  8851222058009799098 | LWLock     | lock_manager   |      4
 2018-07-09 10:44:48.037205+02 |  8221555873158496753 | IO         | DataFileExtend |      1
 2018-07-09 10:44:48.037205+02 | -6860707137622661878 | LWLock     | buffer_content |      4
 2018-07-09 10:45:08.032938+02 |  8851222058009799098 | Lock       | tuple          |      5
 2018-07-09 10:45:08.032938+02 | -6860707137622661878 | Lock       | tuple          |    312
 2018-07-09 10:45:08.032938+02 | -6860707137622661878 | Lock       | transactionid  |   2586
(20 rows)

NOTE: There’s also a per-database version of those metrics for easier computation at the database level in powa_wait_sampling_history_current_db and powa_wait_sampling_history_db

And they are visible in the powa-web interface. Here are some examples of the wait event display with a simple pgbench run:

Wait events for the whole cluster

Wait events for a database

Wait events for a single query

This feature is still under development, but you can already test it using the latest git commits. I hope to add more views of those data in the near future, including some other graphs, since all the data are available. And also, if you’re a python / javascript developer, contributions are always welcome!

Wait Events support for PoWA was originally published by Julien Rouhaud at rjuju's home on July 09, 2018.

Diagnostic of an unexpected slowdown

2018-07-03T17:56:34+00:00

This blog post is a summary of a production issue I had to investigate some time ago with people from Oslandia, and since it’s quite unusual I wanted to share it with some methodology I used, if it can help anyone running into the same kind of problem. It’s also a good opportunity to say that upgrading to a newer PostgreSQL version is almost always a good idea.

The problem

The initial performance issue reported enough information to know something strange was happening.

The database is a PostgreSQL 9.3.5 (yes, missing some minor version updates), and obviously a lot of major versions late. The configuration also had quite unusual settings. The most relevant hardware and settings are:

Server
    CPU: 40 core, 80 with hyperthreading enabled
    RAM: 128 GB
PostgreSQL:
    shared_buffers: 16 GB
    max_connections: 1500

The high shared_buffers, especially given the quite old PostgreSQL version, is a good candidate for more investigation. The max_connections is also quite high, but unfortunately the software vendor claims that using a connection pooler isn’t supported. Therefore most of the connections are idle. This isn’t great because it implies quite some overhead to acquire a snapshot, but there are enough CPU to handle quite a lot of connections.

The main problem was that sometimes, the same queries could be extremely slower. The following trivial example was provided:

EXPLAIN ANALYZE SELECT count(*) FROM pg_stat_activity ;

-- When the issue happens
"Aggregate  (actual time=670.719..670.720 rows=1 loops=1)"
"  ->  Nested Loop  (actual time=663.739..670.392 rows=1088 loops=1)"
"        ->  Hash Join  (actual time=2.987..4.278 rows=1088 loops=1)"
"              Hash Cond: (s.usesysid = u.oid)"
"              ->  Function Scan on pg_stat_get_activity s  (actual time=2.941..3.302 rows=1088 loops=1)"
"              ->  Hash  (actual time=0.022..0.022 rows=12 loops=1)"
"                    Buckets: 1024  Batches: 1  Memory Usage: 1kB"
"                    ->  Seq Scan on pg_authid u  (actual time=0.008..0.013 rows=12 loops=1)"
"        ->  Index Only Scan using pg_database_oid_index on pg_database d  (actual time=0.610..0.611 rows=1 loops=1088)"
"              Index Cond: (oid = s.datid)"
"              Heap Fetches: 0"
"Total runtime: 670.880 ms"

-- Normal timing
"Aggregate  (actual time=6.370..6.370 rows=1 loops=1)"
"  ->  Nested Loop  (actual time=3.581..6.159 rows=1088 loops=1)"
"        ->  Hash Join  (actual time=3.560..4.310 rows=1088 loops=1)"
"              Hash Cond: (s.usesysid = u.oid)"
"              ->  Function Scan on pg_stat_get_activity s  (actual time=3.507..3.694 rows=1088 loops=1)"
"              ->  Hash  (actual time=0.023..0.023 rows=12 loops=1)"
"                    Buckets: 1024  Batches: 1  Memory Usage: 1kB"
"                    ->  Seq Scan on pg_authid u  (actual time=0.009..0.014 rows=12 loops=1)"
"        ->  Index Only Scan using pg_database_oid_index on pg_database d  (actual time=0.001..0.001 rows=1 loops=1088)"
"              Index Cond: (oid = s.datid)"
"              Heap Fetches: 0"
"Total runtime: 6.503 ms"

So while the “good” timing is a little but slow (though there are 1500 connections), the “bad” timing is more than 100x slower, for a very simple query.

Another example of a trivial query on production data was provided, but with some more informations. Here’s an anonymized version:

EXPLAIN (ANALYZE, BUFFERS) SELECT some_col
FROM some_table
WHERE some_indexed_col = 'value' AND uppser(other_col) = 'other_value'
LIMIT 1 ;

"Limit  (actual time=7620.756..7620.756 rows=0 loops=1)"
"  Buffers: shared hit=43554"
"  ->  Index Scan using idx_some_table_some_col on some_table  (actual time=7620.754..7620.754 rows=0 loops=1)"
"        Index Cond: ((some_indexed_cold)::text = 'value'::text)"
"        Filter: (upper((other_col)::text) = 'other_value'::text)"
"        Rows Removed by Filter: 17534"
"        Buffers: shared hit=43554"
"Total runtime: 7620.829 ms"

"Limit  (actual time=899.607..899.607 rows=0 loops=1)"
"  Buffers: shared hit=43555"
"  ->  Index Scan using idx_some_table_some_col on some_table  (actual time=899.605..899.605 rows=0 loops=1)"
"        Index Cond: ((some_indexed_cold)::text = 'value'::text)"
"        Filter: (upper((other_col)::text) = 'other_value'::text)"
"        Rows Removed by Filter: 17534"
"        Buffers: shared hit=43555"
"Total runtime: 899.652 ms"

There was also quite some instrumentation data on O/S side, showing that neither the disk, CPU or RAM where exhausted, and no interesting message in dmesg or any system log.

What do we know?

For the first query, we see that the inner index scan average time raises from 0.001ms to 0.6ms:

->  Index Only Scan using idx on pg_database (actual time=0.001..0.001 rows=1 loops=1088)

->  Index Only Scan using idx on pg_database (actual time=0.610..0.611 rows=1 loops=1088)

With a high shared_buffers setting and an old PostgreSQL version, a common issue is a slowdown when the dataset is larger that the shared_buffers, due to the clocksweep algorithm used to evict buffers.

However, the second query shows that the same thing is happening while all the blocks are in shared_buffers. This cannot be a buffer eviction problem due to too high shared_buffers setting, or any disk latency issue.

While some PostgreSQL configuration settings could be improved, none of them seems to explain this exact behavior. It’d be likely that modifying them will fix the situation, but we need more information to undetstand what’s happening here exactly and avoid any further performance issue.

Any wild guess?

Since the simple explanations have been discarded, it’s necessary to think about lower level explanations.

If you followed the latest PostgreSQL versions enhancements, you should have noticed quite a few optimizations on scalability and locking. If you want more information, there are plenty of blog entries about these, for instance this great article.

On the kernel side and given the high number of connections, it also can be, and it’s probably the most likely explanation, a TLB exhaustion.

In any case, in order to confirm any theory we need to use very specific tools.

Deeper analysis: TLB exhaustion

Without going into too much detail, you need to know that each processus has an area of kernel memory used to store the page tables entries, called the PTE, which are mapping between the virtual addresses that the process are using and the real physical address in RAM. This area is usually not big, because usually a process doesn’t access to gigabytes of data in RAM. But since PostgreSQL is relying on multiple processes accessing a big chunk of shared memory, each process will have an entry for each 4kB (the default page size) address of the shared buffers it has accessed. So you can end up with quite a lot of memory used for the PTE, and even have overall mappings that address way more than the total physical memory available on the server.

You can know the size of the PTE at the O/S level looking for the VmPTE entry in the processus status. You can also check the RssShmem entry to know how many shared memory pages is mapped. For instance:

egrep "(VmPTE|RssShmem)" /proc/${PID}/status
RssShmem:	     340 kB
VmPTE:	     140 kB

This process didn’t access lots of buffers, so the PTE is small. If we try with a process which has accessed all the buffers of a 8 GB shared_buffers:

egrep "(VmPTE|RssShmem)" /proc/${PID}/status
RssShmem:	 8561116 kB
VmPTE:	   16880 kB

It’s 16 MB used for the PTE! Multiplying that with the number of connections, and you end up with gigabytes of memory used for the PTE. Obviously, this wont’ fit in the TLB. As a consequence, the processes will have a lot of TLB miss every time they need to access a page in memory, drastically increasing the latency.

On the system that had performance issue, with 16 GB of shared buffers and 1500 long lived connections, the total memory size of the combined PTE was around 45 GB! A rough approximation can be done with this small script:

for p in $(pgrep postgres); do grep "VmPTE:" /proc/$p/status; done | awk '{pte += $2} END {print pte / 1024 / 1024}'

NOTE: This will compute the memory used for the PTE of all postgres processes. If you have multiple clusters on the same machine and you want to have per cluster information, you need to adapt this command to only match the processes whose ppid are you cluster’s postmaster pid.

This is evidently the culprit here. Just to be sure, let’s look at what perf show when the performance slowdown occurs, and when it doesn’t.

Here are the top consuming functions (consuming more than 2% of CPU time) reported by perf when everything is fine:

# Children      Self  Command          Symbol
# ........  ........  ...............  ..................
     4.26%     4.10%  init             [k] intel_idle
     4.22%     2.22%  postgres         [.] SearchCatCache

Nothing really interesting, the system was really not saturated. Now when the problem occurs:

# Children      Self  Command          Symbol
# ........  ........  ...............  ....................
     8.96%     8.64%  postgres         [.] s_lock
     4.50%     4.44%  cat              [k] smaps_pte_entry
     2.51%     2.51%  init             [k] poll_idle
     2.34%     2.28%  postgres         [k] compaction_alloc
     2.03%     2.03%  postgres         [k] _spin_lock

We can see s_lock, the postgres function that wait on a spinlock, consumes almost 9% of the CPU time. But this is PostgreSQL 9.3, and lightweight locks (transient internal locks) were still implented using spin lock (they now use atomic operation). If we look a little bit more about s_lock calls:

     8.96%     8.64%  postgres         [.] s_lock
                   |
                   ---s_lock
                      |
                      |--83.49%-- LWLockAcquire
[...]
                      |--15.59%-- LWLockRelease
[...]
                      |--0.69%-- 0x6382ee
                      |          0x6399ac
                      |          ReadBufferExtended
[...]

99% of s_lock calls are indeed due to lightweight locks. This indicates a general slowdown and high contentions. But this is just a consequence of the real problem, the next top consumer function.

With almost 5% of the CPU time smaps_pte_entry, a kernel function doing the translation for a single entry, shows the problem. This function is supposed to be extremely fast, and shouldn’t even appear in a perf record! It means that very often, when a process wants to access to page in RAM, it has to wait to get the real address. But waiting for an address translation means a lot of pipeline stalls. Processors have longer and longer pipeline, and those stalls totally ruin the benefits of this kind of architecture. As a result, a good proportion of CPU time is simply wasted waiting for addresses. This certainly explain the extreme slowdown, and the lack of high lever counters able to explain such slowdowns.

The solution

Multiple solutions are possible to solve this problem.

The usual answer is to ask PostgreSQL to allocate the shared_buffers in huge pages. Indeed, with 2MB pages instead of 4kB, the memory needed for PTE will automatically drop 512 times. This would be an easy and huge win. Unfortunately, this is only possible since version 9.4, but upgrading wasn’t even an option, since the software vendor claimed that only the 9.3 version is supported.

Another way to reduce the PTE size is to reduce the number of connections, which is quite high here, and would also probably improve performance. Unfortunately again, the vendor claimed that connection poolers aren’t supported, and the customer needed that many connections.

So the only remaining solution was therefore to reduce the shared_buffers. After some tries, the higher value that could be used to avoid the extreme slowdown was 4 GB. Fortunately, PostgreSQL was able to have quite good performance with this size of dedicated cache.

If software vendors read this post, please understand that if people ask for newer PostgreSQL version compatibility, or pooler compatibility, they have very good reasons for that. There are usually very few behavior changes with newer versions, and they’re all documented!

Diagnostic of an unexpected slowdown was originally published by Julien Rouhaud at rjuju's home on July 03, 2018.

Minimizing tuple overhead

2016-09-16T12:03:34+00:00

I hear quite often people being disappointed on how much space PostgreSQL is wasting for each row it stores. I’ll try to show here some tricks to minimize this effect, to allow more efficient storage.

What overhead?

If you don’t have tables with more than few hundred of million of rows, it’s likely that you didn’t have an issue with this.

For each row stored, postgres will store aditionnal data for its own need. This is documented here. The documentation says:

Field	Type	Length	Description
t_xmin	TransactionId	4 bytes	insert XID stamp
t_xmax	TransactionId	4 bytes	delete XID stamp
t_cid	CommandId	4 bytes	insert and/or delete CID stamp (overlays with t_xvac)
t_xvac	TransactionId	4 bytes	XID for VACUUM operation moving a row version
t_ctid	ItemPointerData	6 bytes	current TID of this or newer row version
t_infomask2	uint16	2 bytes	number of attributes, plus various flag bits
t_infomask	uint16	2 bytes	various flag bits
t_hoff	uint8	1 byte	offset to user data

Which is 23 bytes on most architectures (you have either t_cid or t_xvac).

You can see part of these fields in hidden column present on any table by adding them in the SELECT part of a query, or look for negative attribute number in pg_attribute catalog:

# \d test
     Table "public.test"
 Column |  Type   | Modifiers
--------+---------+-----------
 id     | integer |

# SELECT xmin, xmax, id FROM test LIMIT 1;
 xmin | xmax | id
------+------+----
 1361 |    0 |  1

# SELECT attname, attnum, atttypid::regtype, attlen
FROM pg_class c
JOIN pg_attribute a ON a.attrelid = c.oid
WHERE relname = 'test'
ORDER BY attnum;
 attname  | attnum | atttypid | attlen
----------+--------+----------+--------
 tableoid |     -7 | oid      |      4
 cmax     |     -6 | cid      |      4
 xmax     |     -5 | xid      |      4
 cmin     |     -4 | cid      |      4
 xmin     |     -3 | xid      |      4
 ctid     |     -1 | tid      |      6
 id       |      1 | integer  |      4

If you compare to the previous table, you can see than not all of these columns are not stored on disk. Obviously PostgreSQL doesn’t store the table’s oid in each row. It’s added after, while constructing a tuple.

If you want more technical details, you should read take a look at htup_detail.c, starting with TupleHeaderData struct.

How costly is it?

As the overhead is fixed, it’ll become more and more neglictable as the row size grows. If you only store a single int column (4 bytes), each row will need:

23B + 4B = 27B

So, it’s 85% overhead, pretty horrible.

On the other hand, if you store 5 integer, 3 bigint and 2 text columns (let’s say ~80B average), you’ll have:

23B + 5*4B + 3*8B + 2*80B = 227B

That’s “only” 10% overhead.

So, how to minimize this overhead

The idea is to store the same data with less records. How to do that? Aggregating data in arrays. The more records you put in a single array, the less overhead you have. And if you aggregate enough data, you can benefit from transparent compression thanks to the TOAST mechanism

Let’s try with a single 1 integer column table containing 10M rows:

# CREATE TABLE raw_1 (id integer);

# INSERT INTO raw_1 SELECT generate_series(1,10000000);

# CREATE INDEX ON raw_1 (id);

The user data should need 10M * 4B, ie. around 38MB, while this table will consume 348MB. Inserting the data takes around 23 seconds.

NOTE: If you do the maths, you’ll find out that the overhead is slighty more than 32B, not 23B. This is because each block also has some overhead, NULL handling and alignement issue. If you want more information on this, I recommand to see this presentation

Let’s compare with aggregated versions of the same data:

# CREATE TABLE agg_1 (id integer[]);

# INSERT INTO agg_1 SELECT array_agg(i)
FROM generate_series(1,10000000) i
GROUP BY i % 2000000;

# CREATE INDEX ON agg_1 (id);

This will insert 5 elements per row. I’ve done the same test with 20, 100, 200 and 1000 elements per row. Results are below:

NOTE: The size for 1000 element per row is a little higher than lower value. This is because it’s the only one which is big enough to be TOAST-ed, but not big enough to be compressed. We can see a little TOAST overhead here.

So far so good, we can see quite good improvements, both in size and INSERT time even for very small arrays. Let’s see the impact to retrieve rows. I’ll try to retrieve all the rows, then only one row with an index scan (for the tests I’ve used EXPLAIN ANALYZE to minimize the time to represent the data in psql):

# SELECT id FROM raw_1;

# CREATE INDEX ON raw_1 (id);

# SELECT * FROM raw_1 WHERE id = 500;

To properly index this array, we need a GIN index. To get all the values from aggregated data, we need to unnest() the arrays, and to be a little more creative to get a single record:

# SELECT unnest(id) AS id FROM agg_1;

# CREATE INDEX ON agg_1 USING gin (id);

# WITH s(id) AS (
    SELECT unnest(id)
    FROM agg_1
    WHERE id && array[500]
)
SELECT id FROM s WHERE id = 500;

Here’s the chart comparing index creation time and index size:

The GIN index is a little more than twice the btree index, if I add the table size, total size is almost the same as without aggregation. That’s not a big issue since this example is naive, we’ll see later how to avoid using GIN index to keep total size low. Also index is way slower to build, meaning that INSERT will also be slower.

Here’s the chart comparing the time to retrieve all rows and a single row:

Getting all the rows is probably not an interesting example, but it’s interesting to note that as soon as array contains enough elements it starts to be faster than the same operation using the original table. We also see that getting only one element is much more faster than with the btree index, thanks to GIN efficiency. It’s not tested here, but since only btree index are sorted, if you need to get a lot of data sorted, using a GIN index will require an extra sort which will be way slower than a simple btree index scan.

A more realistic example

Now that we’ve seen the basics, let’s see how to go further: aggregating more than one columns and avoid to use too much disk space (and slowdown at write time) with a GIN index. For this, I’ll present how PoWA stores it’s data.

For each datasource collected, two tables are used: one for the historic and aggregated data, and one the current data. These tables store data in a custom type instead of plain columns. Let’s see the tables related to pg_stat_statements:

The custom type, basically all the counters present in pg_stat_statements and the timestamp associated to this record:

powa=# \d powa_statements_history_record
   Composite type "public.powa_statements_history_record"
       Column        |           Type           | Modifiers
---------------------+--------------------------+-----------
 ts                  | timestamp with time zone |
 calls               | bigint                   |
 total_time          | double precision         |
 rows                | bigint                   |
 shared_blks_hit     | bigint                   |
 shared_blks_read    | bigint                   |
 shared_blks_dirtied | bigint                   |
 shared_blks_written | bigint                   |
 local_blks_hit      | bigint                   |
 local_blks_read     | bigint                   |
 local_blks_dirtied  | bigint                   |
 local_blks_written  | bigint                   |
 temp_blks_read      | bigint                   |
 temp_blks_written   | bigint                   |
 blk_read_time       | double precision         |
 blk_write_time      | double precision         |

The table for current data stores the pg_stat_statement unique identifier (queryid, dbid, userid), and a record of counters:

powa=# \d powa_statements_history_current
    Table "public.powa_statements_history_current"
 Column  |              Type              | Modifiers
---------+--------------------------------+-----------
 queryid | bigint                         | not null
 dbid    | oid                            | not null
 userid  | oid                            | not null
 record  | powa_statements_history_record | not null

The table for aggregated data contains the same unique identifier, an array of records and some special fields:

powa=# \d powa_statements_history
            Table "public.powa_statements_history"
     Column     |               Type               | Modifiers
----------------+----------------------------------+-----------
 queryid        | bigint                           | not null
 dbid           | oid                              | not null
 userid         | oid                              | not null
 coalesce_range | tstzrange                        | not null
 records        | powa_statements_history_record[] | not null
 mins_in_range  | powa_statements_history_record   | not null
 maxs_in_range  | powa_statements_history_record   | not null
Indexes:
    "powa_statements_history_query_ts" gist (queryid, coalesce_range)

We also store the timestamp range (coalesce_range) containing all aggregated counters in the row, and the minimum and maximum values of each counter in two dedicated records. These extra fields doesn’t consume too much space, and allows very efficient indexing and computation, based on the data access pattern of the related application.

This table is used to know how much ressource a query consumed on a given time range. The GiST index won’t be too big since it only indexes two small values per X aggregated counters, and will find efficiently the rows matching a given queryid and time range.

Then, computing the resources consumed can be done efficiently, since the pg_stat_statements counters are strictly monotonic. The algorithm would be:

if the row time range is entirely contained in the asked time range, we only need to compute delta of summary record: maxs_in_range.counter - mins_in_range.counter
if not (meaning only two rows for each queryid) we unnest the array, filter out records that aren’t in the asked time range, keep first and last value and compute for each counter the maximum minus the minimum.

NOTE: Actually, PoWA interface always unnest all records contained in the asked time interval, since the interface is designed to show these counters evolution on a relatively small time range, but with a great precision. Hopefuly, unnesting the arrays is not that expensive, especially compared to the disk space saved.

And here’s the size needed for the aggregated and non aggregated values. For this I let PoWA generate 12.331.366 records (configuring a snapshot every 5 seconds for some hours, with default aggregation of 100 records per row), and used a btree index on (queryid, ((record).ts) to simulate the index present on the aggregated table:

Pretty efficient, right?

Limitations

There are some limitations with aggregating records. If you do this, you can’t enforce constraints such as foreign keys or unique constraints. The use is therefore non-relationnal data, such as counters or metadata.

Bonus

Using custom types also allows some nice things, like defining custom operators. For instance, the release 3.1.0 of PoWA provides two operators for each custom type defined:

the - operator, to get difference between two record
the / operator, to get the difference per second

You can therefore do quite easily this kind of queries:

# SELECT (record - lag(record) over()).*
FROM from powa_statements_history_current
WHERE queryid = 3589441560 AND dbid = 16384;
      intvl      | calls  |    total_time    |  rows  | ...
-----------------+--------+------------------+--------+ ...
 <NULL>          | <NULL> |           <NULL> | <NULL> | ...
 00:00:05.004611 |   5753 | 20.5570000000005 |   5753 | ...
 00:00:05.004569 |   1879 | 6.40500000000047 |   1879 | ...
 00:00:05.00477  |  14369 | 48.9060000000006 |  14369 | ...
 00:00:05.00418  |      0 |                0 |      0 | ...

# SELECT (record / lag(record) over()).*
FROM powa_statements_history_current
WHERE queryid = 3589441560 AND dbid = 16384;

  sec   | calls_per_sec | runtime_per_sec  | rows_per_sec | ...
--------+---------------+------------------+--------------+ ...
 <NULL> |        <NULL> |           <NULL> |       <NULL> | ...
      5 |        1150.6 |  4.1114000000001 |       1150.6 | ...
      5 |         375.8 | 1.28100000000009 |        375.8 | ...
      5 |        2873.8 | 9.78120000000011 |       2873.8 | ...

If you’re interested on how to implement such operators, you can look at PoWA implementation.

Conclusion

You now know the basics to work around the per tuple overhead. Depending on your needs and your data specifities, you should find a way to aggregate your data, maybe add some extra columns, to keep nice performance and spare some disk space.

Minimizing tuple overhead was originally published by Julien Rouhaud at rjuju's home on September 16, 2016.

Estimating Needed Memory for a Sort

2015-08-18T14:03:34+00:00

work_mem?

The work memory, or work_mem is one of the most complicated setting to configure. It can be used for various purposes. It’s mainly used when sorting data or creating hash tables, but it can also be used by set returning functions using a tuplestore for instance, like the generate_series() function. Moreover, each node of a query can use this amount of memory. Set this parameter too low, and a lot of temporary files will be used, set it too high and you may encounter errors, or even an Out Of Memory (OOM) depending on your OS configuration.

I’ll focus here on the amount of memory needed when sorting data, to help you understand how much memory is required when PostgreSQL runs a sort operation.

Truth is out

I sometimes hear people think that there is a correlation between the size of the temporary files generated and the amount of memory that would have been needed to perform the same sort entirely in memory. It’s unfortunately wrong, you can’t make any assumption on the value of work_mem based only on the size of a sort temporary file.

It’s because when the data to be sorted don’t fit in the allowed memory, PostgreSQL will use different algorithms, either external sort or external merge, which have a totally different space usage. In addition to work_mem usage, a smaller temporary file can be used multiple times, with external merge algorithm, for less disk usage and better performance. If you want more details on this, the relevant source code is present in tuplesort.c and logtapes.c. As a brief introduction, the header of tuplesort.c says:

[…] This module handles sorting of heap tuples, index tuples, or single Datums (and could easily support other kinds of sortable objects, if necessary). It works efficiently for both small and large amounts of data. Small amounts are sorted in-memory using qsort(). Large amounts are sorted using temporary files and a standard external sort algorithm.

See Knuth, volume 3, for more than you want to know about the external sorting algorithm. We divide the input into sorted runs using replacement selection, in the form of a priority tree implemented as a heap (essentially his Algorithm 5.2.3H), then merge the runs using polyphase merge, Knuth’s Algorithm 5.4.2D. The logical “tapes” used by Algorithm D are implemented by logtape.c, which avoids space wastage by recycling disk space as soon as each block is read from its “tape”. […]

NOTE: It’s an extract from the 9.5 version of the //readme//. External sorts are now using a //quicksort// quicksort algorithm rather than a //replacement selection//.

It can be easily verified. First, let’s create a table and add some data:

rjuju=# CREATE TABLE sort(id integer, val text);
CREATE TABLE
rjuju=# INSERT INTO sort SELECT i, 'line ' || i
FROM generate_series(1,100000) i;

INSERT 0 100000

To sort all these rows, 7813kB is needed (more details later). Let’s see the EXPLAIN ANALYZE with work_mem set to 7813kB and 7812kB:

rjuju=# SET work_mem to '7813kB';
SET
rjuju=# EXPLAIN ANALYZE SELECT * FROM sort ORDER BY id;
                                                    QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
 Sort  (cost=9845.82..10095.82 rows=100000 width=14) (actual time=50.957..59.163 rows=100000 loops=1)
   Sort Key: id
   Sort Method: quicksort  Memory: 7813kB
   ->  Seq Scan on sort  (cost=0.00..1541.00 rows=100000 width=14) (actual time=0.012..19.789 rows=100000 loops=1)

rjuju=# SET work_mem to '7812kB';
SET

rjuju=# EXPLAIN ANALYZE SELECT * FROM sort ORDER BY id;
                                                    QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
 Sort  (cost=9845.82..10095.82 rows=100000 width=14) (actual time=142.662..168.596 rows=100000 loops=1)
   Sort Key: id
   Sort Method: external sort  Disk: 2432kB
   ->  Seq Scan on sort  (cost=0.00..1541.00 rows=100000 width=14) (actual time=0.027..18.621 rows=100000 loops=1)

So, 7813kB are needed, but if we lack only 1kB, the temporary file size is 2432kB.

You can also activate the trace_sort parameter to have some more information:

rjuju=# SET trace_sort TO on;
SET
rjuju=# SET client_min_messages TO log;
SET

rjuju=# EXPLAIN ANALYZE SELECT * FROM sort ORDER BY id;
LOG:  begin tuple sort: nkeys = 1, workMem = 7812, randomAccess = f
LOG:  switching to external sort with 28 tapes: CPU 0.00s/0.05u sec elapsed 0.05 sec
LOG:  performsort starting: CPU 0.00s/0.07u sec elapsed 0.07 sec
LOG:  finished writing final run 1 to tape 0: CPU 0.00s/0.15u sec elapsed 0.15 sec
LOG:  performsort done: CPU 0.00s/0.15u sec elapsed 0.15 sec
LOG:  external sort ended, 304 disk blocks used: CPU 0.00s/0.18u sec elapsed 0.19 sec
                                                    QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
 Sort  (cost=9845.82..10095.82 rows=100000 width=14) (actual time=154.751..181.724 rows=100000 loops=1)
   Sort Key: id
   Sort Method: external sort  Disk: 2432kB
   ->  Seq Scan on sort  (cost=0.00..1541.00 rows=100000 width=14) (actual time=0.039..23.712 rows=100000 loops=1)

With these data, 28 tapes are used.

So, how do I know how much work_mem is needed?

First, you need to know that all the data will be allocated through PostgreSQL’s allocator AllocSet. If you want to know more about it, I recommend to read the excellent articles Tomas Vondras wrote on this topic: Introduction to memory contexts, Allocation set internals and palloc overhead examples.

The needed information here is that the allocator adds some overhead. Each allocated block has a fixed overhead of 16B, and the memory size requested (without the 16B overhead) will be rounded up to a 2^N size. So if you ask for 33B, 80B will be used: 16B of overhead and 64B, the closest 2^N multiple. The work_mem will be used to store every row, and some more information.

For each row to sort, a fixed amount of 24B memory will be used. This is the size of a SortTuple which is the structure sorted. This amount of memory will be allocated in a single block, so we have only 24B overhead (fixed 8B and the 16B to go to the closest 2^N multiple).

The first part of the formula is therefore:

24 * n + 24

(n being the number of tuple sorted)

Then, you have to know that PostgreSQL will preallocate this space for 1024 rows. So you’ll never see a memory consumption of 2 or 3kB.

Then, each SortTuple will then contain a MinimalTuple, which is basically a tuple without the system metadata (xmin, xmax…), or an IndexTuple if the tuples come from an index scan. This structure will be allocated separately for each tuple, so there can be a pretty big overhead. Theses structures lengths are both 6B, but need to be aligned. This represents 16B per tuple.

These structures will also contain the entire row, the size depends on the table, and the content for variable length columns.

The second part of the formula is therefore:

(8 + ( (16 + average row length) rounded to 2^N) ) * n

We can now estimate how much memory is needed:

(24 + 8 + ( (16 + average row length) rounded to 2^N) ) * n + 24

Testing the formula

Let’s see on our table. It contains two fields, id and val. id is an integer, so it uses 4B. The val column is variable length. First, figure out the estimated average row size:

rjuju=# SELECT stawidth
FROM pg_statistic WHERE starelid = 'sort'::regclass AND staattnum = 2;
 stawidth
----------
       10

Just to be sure, as I didn’t do any ANALYZE on the table:

rjuju=# SELECT avg(length(val)) FROM sort;
        avg
--------------------
 9.8889500000000000

So, the average row size is approximatively 14B. PostgreSQL showed the same estimation on the previous EXPLAIN plan, the reported width was 14:

Sort  (cost=9845.82..10095.82 rows=100000 width=14) [...]

NOTE: It’s better to rely on the pg_statistic, because it’s faster and doesn’t consume resources. Also, if you have large fields, they’ll be toasted, and only a pointer will be stored in work_mem, not the entire field

We add the 16B overhead for the MinimalTuple structure and get 30B. This will lead to an allocated space of 32B.

Finally, the table contains 100.000 tuples, we can now compute the memory needed :

    (24 + 16 + 8 + 32) * 100000 + 24 = 8000024B = 7812,52kB

We now find the 7813kB I announced earlier!

This is a very simple example. If you only sort some of the rows, the estimated size can be too high or too low if the rows you sort don’t match the average size.

Also, note that if the data length of a row exceed 8kB (not counting the toasted data), the allocated size won’t be rounded up to the next 2^N multiple.

Wait, what about NULLs?

Yes, this formula was way too simple…

The formula assume you don’t have any NULL field, so it compute the maximum estimated memory needed.

A NULL field won’t consume space for data, obviously, but will add a bit in a bitmap stored in the MinimalTuple.

If at least one field of a tuple is NULL, the bitmap will be created. Its size is:

(number of attribute + 7) / 8) bytes (rounded down)

So, if a tuple has 3 integer fields, and two of them are NULL, the data size will not be 16B but:

4 + ( (3+7) / 8) = 5B

You can then try to estimate a better size with the statistic NULL fractions of each attribute, available in pg_statistics.

For the lazy ones

Here’s a simple query that will do the maths for you. It assumes:

only fields from one table is sorted
there are no NULL
all the rows will be sorted
statistics are accurate

WITH RECURSIVE overhead(n) AS (
    SELECT 1
    UNION ALL
    SELECT n*2 FROM overhead
    WHERE n <= 4096
),
width AS (
    SELECT starelid,sum(stawidth) AS sum
    FROM pg_statistic
    GROUP BY 1
),
num_of_lines AS (
    SELECT relid,n_live_tup as n
    FROM pg_stat_user_tables

)

SELECT pg_size_pretty(((24 + 16 + 8 + max(o.n)*2) * (min(nol.n))) + 24)
FROM overhead o
CROSS JOIN pg_class c
JOIN pg_namespace n ON c.relnamespace = n.oid
JOIN width w ON w.starelid = c.oid
JOIN num_of_lines nol ON nol.relid = c.oid
WHERE
c.relname = 'sort'
AND n.nspname = 'public'
AND o.n < (w.sum + 16);
 pg_size_pretty
----------------
 7813 kB

Conclusion

Now, you know the basics to estimate the amount of memory you need to sort your data.

A minimal example was presented here for a better understanding, things start to get really complicated when you don’t only sort all the rows of a single table but the result of some joins and filters.

I hope you’ll have fun tuning work_mem on your favorite cluster. But don’t forget, work_mem is used for more than just sorting tuples!

Estimating Needed Memory for a Sort was originally published by Julien Rouhaud at rjuju's home on August 18, 2015.

Keep an eye on your PostgreSQL configuration

2015-07-22T10:48:16+00:00

Have you ever wished to know what configuration changed during the last weeks, when everything was so much faster, or wanted to check what happened on your beloved cluster while you were in vacation?

pg_track_settings is a simple, SQL only extension that helps you to know all of that and more very easily. As it’s designed as an extension, it requires PostgreSQL 9.1 or more.

Some insights

As amost any extension, you have to compile it from source, or use the pgxn client, since there’s no package yet. Assuming you just extract the tarball of the release 1.0.0 with a typical server configuration:

$ cd pg_track_settings-1.0.0
$ sudo make install

Then the extension is available. Create the extension on the database of your choice:

postgres=# CREATE EXTENSION pg_track_settings ;
CREATE EXTENSION

In order to historize the settings, you need to schedule a simple function call on a regular basis. This function is the pg_track_settings_snapshot function. It’s really cheap to call, and won’t have any measurable impact on your cluster. This function will do all the smart work of storing all the parameters that changed since the last call.

For instance, if you want to be able to know what changed on your server within a 5 minutes accuracy, a simple cron entry like this for the postgres user is enough:

*/5 *  * * *     psql -c "SELECT pg_track_settings_snapshot()" > /dev/null 2>&1

A background worker could be used on PostgreSQL 9.3 and more, but as we only have to call one function every few minutes, it’d be overkill to add one just for this. If you really want one, you’d better consider settting up PoWA for that, or another extension that allows to run task like pgAgent.

How to use it

Let’s call the snapshot function to get ti initial values:

postgres=# select pg_track_settings_snapshot()
 ----------------------------
  t
  (1 row)

A first snapshot with the initial settings values is saved. Now, I’ll just change a setting in the postgresql.conf file (ALTER SYSTEM could also be used on a PostgreSQL 9.4 or more release), reload the configuration and take another snapshot:

postgres=# select pg_reload_conf();
 pg_reload_conf
 ----------------
  t
  (1 row)

postgres=# select * from pg_track_settings_snapshot();
 pg_track_settings_snapshot
----------------------------
 t
(1 row)

Now, the fun part. What information is available?

First, what changed between two timestamp. For instance, let’s check what changed in the last 2 minutes:

postgres=# SELECT * FROM pg_track_settings_diff(now() - interval '2 minutes', now());
        name         | from_setting | from_exists | to_setting | to_exists
---------------------+--------------|-------------|------------|----------
 max_wal_size        | 93           | t           | 31         | t
(1 row)

What do we learn ?

as the max_wal_size parameter exists, I’m using the 9.5 alpha release. Yes, what PostgreSQL really needs right now is people testing the upcoming release! It’s simple, and the more people test it, the faster it’ll be avalable. See the how to page to see how you can help :)
the max_wal_size parameter existed 2 minutes ago (from_exists is true), and also exists right now (to_exists is true). Obviously, the regular settings will not disappear, but think of extension related settings like pg_stat_statements.* or auto_explain.*
the max_wal_size changed from 93 (from_setting) to 31 (to_setting).

Also, we can get the history of a specific setting:

postgres=# SELECT * FROM pg_track_settings_log('max_wal_size');
              ts               |     name     | setting_exists | setting
-------------------------------+--------------+----------------+---------
 2015-07-17 22:42:01.156948+02 | max_wal_size | t              | 31
 2015-07-17 22:38:02.722206+02 | max_wal_size | t              | 93
(2 rows)

You can also retrieve the entire configuration at a specified timestamp. For instance:

postgres=# SELECT * FROM pg_track_settings('2015-07-17 22:40:00');
                name                 |     setting
-------------------------------------+-----------------
[...]
 max_wal_senders                     | 5
 max_wal_size                        | 93
 max_worker_processes                | 8
[...]

The sames functions are provided to know what settings have been overloaded for a specific user and/or database (the ALTER ROLE … SET, ALTER ROLE … IN DATABASE … SET and ALTER DATABASE … SET commands), with the functions:

pg_track_db_role_settings_diff()
pg_track_db_role_settings_log()
pg_track_db_role_settings()

And finally, just in case you can also know when PostgreSQL has been restarted:

postgres=# SELECT * FROM pg_reboot;
              ts
-------------------------------
 2015-07-17 08:39:37.315131+02
(1 row)

That’s all for this extension. I hope you’ll never miss or forget a configuration change again!

If you want to install it, the source code is available on the github repository github.com/rjuju/pg_track_settings.

Limitations

As the only way to know what is the current value for a setting is to query pg_settings (or call current_setting()), you must be aware that the user calling pg_track_settings_snapshot() may see an overloaded value (like ALTER ROLE … SET param = value) rather than the original value. As the pg_db_role_setting table is also historized, it’s pretty easy to know that you don’t see the original value, but there’s no way to know what the original value really is.

Keep an eye on your PostgreSQL configuration was originally published by Julien Rouhaud at rjuju's home on July 22, 2015.

How About Hypothetical Indexes ?

2015-07-02T10:08:03+00:00

After so much time missing this features, HypoPG implements hypothetical indexes support for PostgreSQl, available as an extension.

Introduction

It’s now been some time since the second version of PoWA has been announced. One of the new feature of this version is the pg_qualstats extension, written by Ronan Dunklau.

Thanks to this extension, we can now gather real-time statistics to detect missing indexes, and much more (if you’re interested in this extension, you should read Ronan’s article about pg_qualstats). And used with PoWA, you have an interface that allows you to find the most consuming queries, and will suggest you the missing indexes if they’re needed.

That’s really nice, but now a lot of people come with this natural question: Ok, you say that I should create this index, but will PostgreSQL eventually use it ?. That’s actually a good question, because depending on many parameters (in many other things), PostgreSQL could choose to just ignore your freshly created index. That could be a really bad surprise, especially if you had to wait many hours to have it built.

Hypothetical Indexes

So yes, the answer to this question is hypothetical indexes support. That’s really not a new idea, a lot of popular RDBMS support them.

There has already been some previous work on this several years ago, presented at pgCon 2010, which was implementing much more than hypothetical indexes, but this was a research work, which means that we never saw those features coming up in PostgreSQL. This great work is only available as a fork of a few specific PostgreSQL versions, the most recent being 9.0.1.

lightweight implementation: HypoPG

I had quite a different approach in HypoPG to implement hypothetical indexes support.

first of all, it must be completely pluggable. It’s available as an extension and can be used (for now) on any 9.2 or higher PostgreSQL server.
it must be as non intrusive as it’s possible. It’s usable as soon as you create the extension, without restart. Also, each backend has its own set of hypothetical indexes, which mean that adding an hypothetical index will not disturb other connections. Also, the hypothetical indexes are stored in memory, adding/removing a huge amount of them will not bloat your system catalog.

The only restriction in implementing such a feature as an extension is that you can’t change the syntax without modifying the PostgreSQL source code. So, everything has to be done through user defined functions, and change regular behaviour of existing functionnalities, like the EXPLAIN command. We’ll study the details later.

Features

For now, the following functions are available:

hypopg(): return the list of hypothetical indexes (in a similar way as pg_index).
hypopg_add_index(schema, table, attribute, access_method): create a 1-column-only hypothetical index.
hypopg_create_index(query): create an hypothetical index using a standard CREATE INDEX statement.
hypopg_drop_index(oid): remove the specified hypothetical index.
hypopg_list_indexes(): return a short human readable version list of available hypothetical indexes.
hypopg_relation_size(oid): return the estimated size of an hypothetical index
hypopg_reset(): remove all hypothetical indexes

If some hypothetical indexes exists for some relations used in an EXPLAIN (without ANALYZE) statement, they will automatically be added to the list of real indexes. PostgreSQL will then choose to use them or not.

Usage

Installing HypoPG is quite simple. Assuming you downloaded and extracted a tarball in the hypopg-0.0.1 directory, are using a packaged version of PostgreSQL and have -dev packages:

$ cd hypopg-0.0.1
$ make
$ sudo make install

Then HypoPG should be available:

rjuju=# CREATE EXTENSION hypopg ;
CREATE EXTENSION

Let’s try some really simple tests. First, create a small table:

rjuju=# CREATE TABLE testable AS SELECT id, 'line ' || id val
rjuju=# FROM generate_series(1,1000000) id;
SELECT 100000
rjuju=# ANALYZE testable ;
ANALYZE

Then, let’s see a query plan that should benefit an index that’s not here:

rjuju=# EXPLAIN SELECT * FROM testable WHERE id < 1000 ;
                          QUERY PLAN
---------------------------------------------------------------
 Seq Scan on testable  (cost=0.00..17906.00 rows=916 width=15)
   Filter: (id < 1000)
(2 rows)

No surprise, a sequential scan is the only way to go. Now, let’s try to add an hypothetical index, and EXPLAIN again:

rjuju=# SELECT hypopg_create_index('CREATE INDEX ON testable (id)');
 hypopg_create_index
---------------------
 t
(1 row)

Time: 0,753 ms

rjuju=# EXPLAIN SELECT * FROM testable WHERE id < 1000 ;
                                          QUERY PLAN
-----------------------------------------------------------------------------------------------
 Index Scan using <41079>btree_testable_id on testable  (cost=0.30..28.33 rows=916 width=15)
   Index Cond: (id < 1000)
(2 rows)

Yeah! Our hypothetical index is used. We also notice that the hypothetical index creation is more or less 1ms, which is way less than the real index creation would have last.

And of course, this hypothetical index is not used in an EXPLAIN ANALYZE:

rjuju=# EXPLAIN ANALYZE SELECT * FROM testable WHERE id < 1000 ;
                                                 QUERY PLAN
-------------------------------------------------------------------------------------------------------------
 Seq Scan on testable  (cost=0.00..17906.00 rows=916 width=15) (actual time=0.076..234.218 rows=999 loops=1)
   Filter: (id < 1000)
   Rows Removed by Filter: 999001
 Planning time: 0.083 ms
 Execution time: 234.377 ms
(5 rows)

Now let’s go further:

rjuju=# EXPLAIN SELECT * FROM testable
rjuju=# WHERE id < 1000 and val LIKE 'line 100000%';
                                         QUERY PLAN
---------------------------------------------------------------------------------------------
 Index Scan using <41079>btree_testable_id on testable  (cost=0.30..30.62 rows=1 width=15)
   Index Cond: (id < 1000)
   Filter: (val ~~ 'line 100000%'::text)
(3 rows)

Our hypothetical index is still used, but an index on id and val should help this query. Also, as there’s a wildcard on the right-side of the LIKE pattern, the operator class text_pattern_ops is needed. Let’s check that:

rjuju=# SELECT hypopg_create_index('CREATE INDEX ON testable (id, val text_pattern_ops)');
 hypopg_create_index
---------------------
 t
(1 row)

Time: 1,194 ms

rjuju=# EXPLAIN SELECT * FROM testable
rjuju=# WHERE id < 1000 and val LIKE 'line 100000%';
                                              QUERY PLAN
------------------------------------------------------------------------------------------------------
 Index Only Scan using <41080>btree_testable_id_val on testable on testable  (cost=0.30..26.76 rows=1 width=15)
   Index Cond: ((id < 1000) AND (val ~>=~ 'line 100000'::text) AND (val ~<~ 'line 100001'::text))
   Filter: (val ~~ 'line 100000%'::text)
(3 rows)

And yes, PostgreSQL decides to use our new index!

Index size estimation

For now, the index size estimation is done quickly, which can give us a clue on what would be the real index size.

Let’s check the estimated size of our two hypothetical indexes:

rjuju=# SELECT indexname,pg_size_pretty(hypopg_relation_size(indexrelid))
rjuju=# FROM hypopg();
           indexname           | pg_size_pretty 
-------------------------------+----------------
 <41080>btree_testable_id     | 25 MB
 <41079>btree_testable_id_val | 49 MB
(2 rows)

Now, create the real indexes, and compare the sizes:

rjuju=# CREATE INDEX ON testable (id);
CREATE INDEX
Time: 1756,001 ms

rjuju=# CREATE INDEX ON testable (id, val text_pattern_ops);
CREATE INDEX
Time: 2179,185 ms

rjuju=# SELECT relname,pg_size_pretty(pg_relation_size(oid))
rjuju=# FROM pg_class WHERE relkind = 'i' AND relname LIKE '%testable%';
       relname       | pg_size_pretty 
---------------------+----------------
 testable_id_idx     | 21 MB
 testable_id_val_idx | 30 MB

The estimated index size is a bit higher than the real size. It’s on purpose. If the estimated index size is less than an existing index, PostgreSQL would prefer to use the hypothetical index than the real index, which is definitively not interesting. Also, to simulate a bloated index (which is quite frequent on real indexes), a hardcoded 20% bloat factor is added. Finally, the estimation could also be improved a lot.

Limitations

This 0.0.1 version of HypoPG is still a work in progress, and a lot of work is still needed.

Here are the main limitations (at least that I’m aware of):

only btree hypothetical indexes are supported
no hypothetical indexes on expression
no hypothetical indexes on predicate
tablespace specification is not possible
index size estimation could be improved, and it’s not possible to change the bloat factor

However, I believe it can already be helpful.

What’s next ?

Now, the next step is to implement HypoPG support in PoWA, to help DBA decide wether they should create the suggested index or not, and remove the current limitations.

If you want to try HypoPG, here is the github repository: github.com/HypoPG/hypopg.

Stay tuned!

How About Hypothetical Indexes ? was originally published by Julien Rouhaud at rjuju's home on July 02, 2015.

Talking About OPM and PoWA at pgconf.ru

2015-03-18T08:09:10+00:00

Last month, I had the chance to talk about PostgreSQL monitoring, and present some of the tools I’m working on at pgconf.ru.

This talk was a good opportunity to work on an overview of existing projects dealing with monitoring or performance, see what may be lacking and what can be done to change this situation.

Here are my slides:

If you’re interested in this topic, or if you developped a tool I missed while writing these slides (my apologies if it’s the case), the official wiki page is the place you should go first.

I’d also like to thank all the pgconf.ru staff for their work, this conference was a big success, and the biggest postgresql-centric event ever organized.

Talking About OPM and PoWA at pgconf.ru was originally published by Julien Rouhaud at rjuju's home on March 18, 2015.