Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
176228e
Add ds.py wrapper, eager round-trip foundation (Phase 1)
ghostiee-11 May 12, 2026
0a8b15a
Add lazy SQLBackendArray, default to lazy round-trip (Phase 2)
ghostiee-11 May 12, 2026
18c6d1c
Sparse-extent handling and edge cases (Phase 3)
ghostiee-11 May 13, 2026
36dfb4b
Document SQL -> xarray round-trip in README (Phase 4)
ghostiee-11 May 13, 2026
479133c
Optimize lazy memory: skip DISTINCT scans + omit redundant WHERE clauses
ghostiee-11 May 13, 2026
f50ba86
Tighten type annotations and cover vectorized indexer path
ghostiee-11 May 13, 2026
ade544a
Rearchitect lazy backend on DataFusion DataFrame API + Arrow streaming
ghostiee-11 May 28, 2026
289bedb
Merge remote-tracking branch 'origin/main' into feat/lazy-sql-to-xarray
ghostiee-11 May 28, 2026
50b9f52
fix: changes
ghostiee-11 Jun 6, 2026
7b7518b
fix: address round-4 review
ghostiee-11 Jun 12, 2026
f5f0160
fix : conflicts
ghostiee-11 Jun 12, 2026
ac53d04
Improve performance: execute passes for to_dataframe _once_.
alxmrs Jun 14, 2026
9c48b53
Major fix: gutting the introspection path. Using output chunks as a k…
alxmrs Jun 14, 2026
6d50108
Documented the current feature in the README.
alxmrs Jun 14, 2026
8d58e82
dimension_columns --> dims
alxmrs Jun 14, 2026
f535f3c
Partition-snapped "auto" chunks implementation.
alxmrs Jun 14, 2026
221a28d
rm reverse docs, they can be documented elsewhere. Also, it currently…
alxmrs Jun 14, 2026
3d49c4a
unifiying "template_table" with just the "template" argument.
alxmrs Jun 14, 2026
1a29fa4
chore: ruff format
ghostiee-11 Jun 15, 2026
974b5e3
Attribution.
alxmrs Jun 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

67 changes: 27 additions & 40 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,52 +57,37 @@ ctx.sql('''
# 0 8.640069

# Average temperature per pressure level, globally.
ctx.sql('''
result = ctx.sql('''
SELECT level, AVG(temperature) - 273.15 AS avg_c
FROM era5.atmosphere
WHERE time BETWEEN TIMESTAMP '2020-01-01'
AND TIMESTAMP '2020-01-01 05:00:00'
GROUP BY level
ORDER BY level DESC
''').to_pandas()
# level avg_c
# 0 1000 6.621012 ← surface
# 1 975 5.185638
# 2 950 4.028429
# 3 925 3.082812
# 4 900 2.210917
# 5 875 1.395018
# 6 850 0.634267
# 7 825 -0.210372
# 8 800 -1.181075
# 9 775 -2.306465
# 10 750 -3.535534
# 11 700 -6.241685
# 12 650 -9.236364
# 13 600 -12.580938
# 14 550 -16.335386
# 15 500 -20.643604
# 16 450 -25.573401
# 17 400 -31.156920
# 18 350 -37.400552
# 19 300 -43.852607
# 20 250 -49.322132
# 21 225 -51.569113
# 22 200 -53.693248
# 23 175 -55.890484
# 24 150 -58.382290
# 25 125 -61.091916
# 26 100 -63.624885 ← tropopause
# 27 70 -63.182300
# 28 50 -60.124845
# 29 30 -55.986327
# 30 20 -52.433089
# 31 10 -44.140750
# 32 7 -38.707350
# 33 5 -32.621999
# 34 3 -21.509175
# 35 2 -13.355764
# 36 1 -9.020513 ← top of atmosphere
''')
# DataFrame()
# +-------+----------------------+
# | level | avg_c |
# +-------+----------------------+
# | 1000 | 6.6210120796502565 |
# | 975 | 5.185637919348153 |
# | 950 | 4.028428657263021 |
# | 925 | 3.0828117974912743 |
# | 900 | 2.2109172992531967 |
# | 875 | 1.395017610194202 |
# | 850 | 0.6342670572626616 |
# | 825 | -0.21037158786759846 |
# | 800 | -1.1810754318269687 |
# | 775 | -2.3064649711534457 |
# +-------+----------------------+

avg_temp_ds = result.to_dataset(dims=["level"])
# <xarray.Dataset> Size: 592B
# Dimensions: (level: 37)
# Coordinates:
# * level (level) int64 296B 1000 975 950 925 900 875 850 ... 20 10 7 5 3 2 1
# Data variables:
# avg_c (level) float64 296B 6.621 5.186 4.028 ... -21.51 -13.36 -9.021
```

_(A runnable version of this example lives at
Expand Down Expand Up @@ -211,6 +196,8 @@ I want to give a special thanks to the following folks and institutions:
who are working to make this library better.
- Andrew Huang for the sense of taste he brings to the project and consummate code
changes.
- Aman Kumar for spending a considerable amount of his GSoC internship
contributing to this project.


## License
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ module = [
"pyarrow.*",
"datafusion.*",
"xarray.*",
"pandas.*",
]
ignore_missing_imports = true

Expand Down
Loading
Loading