update: Sail to 0.6.6#982
Conversation
| SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime, SearchPhrase LIMIT 10; | ||
| SELECT CounterID, AVG(length(URL)) AS l, COUNT(*) AS c FROM hits WHERE URL <> '' GROUP BY CounterID HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; | ||
| SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; | ||
| SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '$1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; |
There was a problem hiding this comment.
Query was previously returning incorrect results. Should be $1.
|
|
||
| os.environ["SAIL_PARQUET__BINARY_AS_STRING"] = "true" | ||
| os.environ["SAIL_PARQUET__REORDER_FILTERS"] = "true" | ||
| os.environ["SAIL_OPTIMIZER__ENABLE_JOIN_REORDER"] = "true" |
There was a problem hiding this comment.
This does nothing since ClickBench has no joins
| os.environ["SAIL_PARQUET__BINARY_AS_STRING"] = "true" | ||
| os.environ["SAIL_PARQUET__REORDER_FILTERS"] = "true" | ||
| os.environ["SAIL_OPTIMIZER__ENABLE_JOIN_REORDER"] = "true" |
There was a problem hiding this comment.
This does nothing since ClickBench has no joins
| SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime, SearchPhrase LIMIT 10; | ||
| SELECT CounterID, AVG(length(URL)) AS l, COUNT(*) AS c FROM hits WHERE URL <> '' GROUP BY CounterID HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; | ||
| SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; | ||
| SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '$1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; |
There was a problem hiding this comment.
Query was previously returning incorrect results. Should be $1.
| spark = SparkSession.builder.remote(f"sc://localhost:{port}").getOrCreate() | ||
|
|
||
| df = spark.read.parquet("partitioned") | ||
| df = df.withColumn("EventTime", F.col("EventTime").cast("timestamp")) |
There was a problem hiding this comment.
I added this in #967, turns out it is unnecessary.
| spark = SparkSession.builder.remote(f"sc://localhost:{port}").getOrCreate() | ||
| df = spark.read.parquet("hits.parquet") | ||
| df = df.withColumn("EventTime", F.col("EventTime").cast("timestamp")) |
There was a problem hiding this comment.
I added this in #967, turns out it is unnecessary.
|
ClickBench currently measures Sail 0.6.5. This PR upgrades to 0.6.6, released one week after 0.6.5. Did anything related to performance change in this seemingly minor release? Would it be possible to postpone the next ClickBench submission until the next major release? Asking because creating measurements (like here just a couple of days ago) creates some manual work on our end. |
|
@rschu1ze Yeah we can wait until the big Sail release. I'll leave this PR open for now and ping you when it's updated |
2 participants
No description provided.