MySQL: fix handling of TIMESTAMP when using TEXT COLUMN#36612
MySQL: fix handling of TIMESTAMP when using TEXT COLUMN#36612martykulma wants to merge 5 commits into
Conversation
def-
left a comment
There was a problem hiding this comment.
Running bin/mzcompose --find mysql-cdc down && bin/mzcompose --find mysql-cdc run cdc text-columns-timestamp.td shows me scary temporary retraction errors:
> SELECT id, created_at FROM products ORDER BY id;
executing query failed
Caused by:
0: db error
1: ERROR: Invalid data in source, saw retractions (1) for row that does not exist: [Int32(5), String("1970-01-01 00:00:00")]
rows didn't match; sleeping to see if dataflow catches up 🕑 50ms
rows match; continuing at ts 1779156671.0067012, took 0.949501151s
If we change the handling of the binlog decoding we also have to make the same change in the snapshot code!
Definitely a scary error! I ran this over a dozen times yesterday (against 9.5, 8.4, 8.0), and again for each version this morning, but still no failures. I definitely missed some thing in looking at this yesterday. I just came across https://dev.mysql.com/doc/refman/8.0/en/datetime.html -- and this has got me worried because the behavior or DATETIME and TIMESTAMP are different, but in MZ we don't distinguish -- which is a problem! |
|
I think they are caused by this:
|
Seems like the most likely case (the snapshot case is decoding as Value::Date, where binlog will decode as Value::Bytes). I'm not sure why I could not repro the error locally (seems like it should happen fairly consistently). PR is updated with additional settings. |
When using TEXT COLUMN for a timestamp column, Materialize did not have proper handling for timestamps read from binlog that were using either the legacy (pre-5.6) representation MYSQL_TYPE_TIMESTAMP or the post 5.6 MYSQL_TYPE_TIMESTAMP2.
This resulted in user not being able to ingest invalid timestamps (like MySQL's special "zero" value timestamps "0000-00-00 00:00:00")
Fixes SS-151: MySQL: decode fails for TIMESTAMP column when using TEXT COLUMNS