In embedded, is there any difference between a device driver and a library?

In embedded, is there any difference between a device driver and a library?

Assuming a platform with no kernel mode, such as Atmel AVR, is there any
difference between a device driver and a library, given that everything is
user mode anyway?
I ask because I'm thinking about how I want to layer my code. It's like this:
+----------------+
| Business logic |
+----------------+
| CAN library |
+----------------+
| MCP2515 driver |
+----------------+
| SPI driver |
+----------------+
The SPI "driver" has an interrupt handler, and it talks directly to the
microcontroller's SPI peripheral, which sounds like a driver, but other
than that, I don't see how it's any different to, say, the CAN library,
which has high-level functions like "send this message".