Angular multipart/formdata
You would like to upload or post something not as an application/json but as a multipart/formdata.
It’s pretty simple, simpler than you would’ve thought 🙂
1. create the FormData instance
|
1 2 3 |
public JSON2FormData() { const fd = new FormData(); } |
then you can use .set or .append methods to add values to the post body of the FormData.
|
1 2 3 4 5 6 7 8 9 10 |
public JSON2FormData() { const fd = new FormData(); fd.set('greeting', 'hello there'); // upload a file, as in `File()` // let's assume our component has a `private file: File = new File()` fd.set('file', this.file, 'filename'); // or file.name instead of 'filename' // or you have an object that you'd like to send as a JSON object // let's assume out component has a `private myobject: object = {}` fd.set('myjson', JSON.stringify(this.myobject)); } |
and finally we upload it or send it away, rather.
|
1 2 3 4 |
// some service public uploadFormData(fd: FormData) { return this.http.post('/url', fd); } |
if we have a file upload and would like to be notified about upload progress
|
1 2 3 4 5 6 |
public uploadFormData(fd: FormData) { return this.http.post('/url', fd, { reportProgress: true, observe: 'events', }); } |
when we subscribe to this, we have several events happening.
|
1 2 3 4 5 6 7 8 |
this.formdataService.uploadFormData.subscribe(event => { switch (event.type) { case HttpEventType.UploadProgress: case HttpEventType.ResponseHeader: case HttpEventType.Response: // etc } }) |
Angular, EventSource, Go and wasted lifetime
If you have ever used EventSource and Angular and have read all the other blog posts and github issues and Stackoverflow posts and nothing worked, then you’ve probably come here and will happily leave knowing you solved your problem.
First of all I use cobra, I know, get to the point. As part of cobra I add the serve command, so I can go run main.go serve
This is the serve.go file
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
package cmd import ( "git.icod.de/dalu/eventsrc/server/handler" "github.com/gin-contrib/cors" "github.com/gin-gonic/gin" "github.com/spf13/cobra" ) // serveCmd represents the serve command var serveCmd = &cobra.Command{ Use: "serve", Short: "", Long: ``, RunE: func(cmd *cobra.Command, args []string) error { r := gin.Default() r.Use(cors.Default()) h := handler.NewHandler() defer h.Close() r.GET("/api/v1/events/", h.Stream) return r.Run(":8080") }, } func init() { rootCmd.AddCommand(serveCmd) } |
I set up the route, cors and run it
The server/handler/handler.go file
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 |
package handler import ( "fmt" "log" "time" "github.com/gin-contrib/sse" "github.com/gin-gonic/gin" ) type Handler struct { t *time.Ticker } func NewHandler() *Handler { h := new(Handler) h.t = time.NewTicker(time.Second * 1) return h } func (h *Handler) Close() { h.t.Stop() } func (h *Handler) Stream(cx *gin.Context) { i := 0 w := cx.Writer clientGone := w.CloseNotify() for { select { case <-clientGone: return case t := <-h.t.C: type M struct { Id int `json:"id"` Model string `json:"model"` Action string `json:"action"` Time time.Time `json:"time"` } m := new(M) m.Model = "profile" m.Action = "update" m.Id = 1 m.Time = t h := w.Header() h.Set("Cache-Control", "no-cache") h.Set("Connection", "keep-alive") h.Set("Content-Type", "text/event-stream") h.Set("X-Accel-Buffering", "no") ev := sse.Event{ Id: fmt.Sprintf("%d", i), Event: "message", Data: m, } if e := sse.Encode(w, ev); e != nil { log.Println(e.Error()) return } w.Flush() i++ } } } |
Here is the important part. I wasted the last 6 hours and previous to that 2 days on this issue.
If you’re serving this via nginx, you have to set this header X-Accel-Buffering = no.
If you don’t send this header responses will get buffered by nginx until the timeout it met then flushed to the client.
The above code has a ticker that ticks every second and sends a new “Server Sent Event”.
Why it didn’t work for me was, as you see above Event: "message". I had that set to “darko”.
The Angular service
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
import {Injectable, NgZone} from '@angular/core'; import {Observable} from 'rxjs'; @Injectable({ providedIn: 'root' }) export class NotiService { constructor(private zone: NgZone) { this.zone = new NgZone({ enableLongStackTrace: false }); } watch(): Observable<object> { return Observable.create((observer) => { const eventSource = new EventSource('/api/v1/events/'); eventSource.onmessage = (event) => this.zone.run(() => { console.log(event); observer.next(JSON.parse(event.data)); }); eventSource.addEventListener('darko', (event: any) => this.zone.run(() =>{ console.log('darko event', event); observer.next(JSON.parse(event.data)); })); eventSource.onerror = error => this.zone.run(() => { if (eventSource.readyState === eventSource.CLOSED) { console.log('The stream has been closed by the server.'); eventSource.close(); observer.complete(); } else { observer.error(error); } }); return () => eventSource.close(); }); } } |
eventSource.onmessage expects a message with the Event: "message" content. Since I had it set to “darko”,
the onmessage event never fired. If you for whatever reason need to send an event that is not a message type,
the eventSource.addEventListener is how you listen for that event.
As you might have seen in other blog posts or github issues, zonejs and EventSource aren’t the best of friends.
So you have to wrap it all in zone.run() so you can have real time updates, and not just when you unsubscribe from the Observable.
Finally, the component
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
import {Component, OnDestroy, OnInit} from '@angular/core'; import {NotiService} from '../noti.service'; @Component({ selector: 'icod-home', templateUrl: './home.component.html', styleUrls: ['./home.component.scss'] }) export class HomeComponent implements OnInit, OnDestroy { msgSub; msgs = []; constructor(private notiService: NotiService) { } ngOnInit() { } ngOnDestroy(): void { if (this.msgSub) { this.msgSub.unsubscribe(); } } watchEvents() { this.msgSub = this.notiService.watch().subscribe( data => { console.log(data); this.msgs.push(data); }); } stopWatching() { this.msgSub.unsubscribe(); } } |
and the component html
|
1 2 3 4 5 6 |
<button (click)="watchEvents()">Watch Events</button> <button (click)="stopWatching()">Stop Watching</button> <div *ngFor="let msg of msgs"> {{msg|json}} </div> |
Finally, the nginx configuration for the development server. To serve it all.
Here I’m using es.dev.luketic on the local network.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
server { listen 80; listen [::]:80; server_name es.dev.luketic; root /home/darko/WebProjects/es/src; index index.html; error_log /var/log/nginx/es.error; location / { proxy_pass http://localhost:4200; proxy_read_timeout 30; proxy_connect_timeout 30; proxy_redirect off; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } location /sockjs-node/ { proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; rewrite ^/(.*)$ /$1 break; proxy_set_header Host localhost; proxy_pass http://localhost:4200/; } location ~ ^/api/v1/.* { proxy_pass http://localhost:8080; proxy_read_timeout 30; proxy_connect_timeout 30; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } |
libvirt manual migration
Following scenario:
You have 2 servers. No shared storage.
You would like to move VMs from Server A to Server B.
If you want a clean filesystem there will be downtime.
Downtime will take as long as it needs to copy the image from one server to the other.
Step 1:
Shutdown the instance running on the source server.
e.g.
|
1 |
virsh shutdown domain.name |
Step 2:
Copy the image, usually located in /var/lib/libvirt/images/ , from source to destination.
e.g.
|
1 |
rsync -avh /var/lib/libvirt/images/* root@destination.server:/var/lib/libvirt/images/ |
Step 3:
Dumo the configuration xml of your vm and copy it to the destination server
e.g.
|
1 |
virsh dumpxml domain.name > domain.name.xml |
|
1 |
scp domain.name.xml root@destination.server: |
This will copy domain.name.xml to the home directory of the destination server, in this case /root/ (implied by the : after the server name)
Step 4:
Import the copied xml domain configuration on the destination server.
The MAC address probably changed, depending on your data center’s setup you’d need to edit the xml file accordingly.
The domain import command is
|
1 |
virsh define domain.name.xml |
Step 5:
Finish it up.
Change network configuration from inside the guest.
You will need to login via spice console.
It’s best to use the great tool “Virtual Machine Manager” for this job.
Remember to keep your VM’s root password ready.
Go database changes notifications over websocket. Generic solution. Reactive apps.
Do you know the problem when you’re writing single page apps and you have to manually update your data from the backend because you know you have just made a change and want the current state to update in “real time”? So you either load the data again by hand/initialite a refresh of the data, or you reload the page or you navigate to a redirect component to land back at your page so it’s loaded anew?
Well there are databases who can notify you of changes done. There’s RethinkDB, there’s Postgresql’s NOTIFY and LISTEN, there’s MongoDB’s change stream that never worked for me and I could never receive any help because IRC was always dead when I joined.
Anyhow, instead of relying on database to notify us that changes were made why not just have the changes in our handler or repository?
I wrote
https://git.icod.de/dalu/notifier
a library that deals with the problem.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
func main() { r := gin.Default() m := melody.New() n := notifier.NewNotifier(m) h := NewHandler(n) r.GET("/", h.Index) r.GET("/ws", func(cx *gin.Context) { m.HandleRequest(cx.Writer, cx.Request) }) r.GET("/acme/ins", h.InsertDB) go n.Listen(func(notification *notifier.Notification) { b, e := json.Marshal(notification) if e != nil { log.Println(e.Error()) return } m.Broadcast(b) }) r.Run(":5000") } |
So you need an instance of melody, created with melody.New()
You pass this instance to create a new Notifier instance and you pass this notifier instance to the handler if you use that.
GET / serves the index.html page that’s essentially a copy/paste of melody’s chat app and does connect and receive notifications via WebSocket. I mean that’s what it does, it returns the HTML that does it.
GET /ws is the WebSocket that is used to connect with
Now
|
1 |
go n.Listen() |
with the paramter
|
1 |
func(notification *notifier.Notification) {} |
That parameter is the callback function that is called when a notification is sent.
In this case the notification is marshalled into []byte and then broadcast by melody to all connected WebSockets.
Later in your handler, in this example we only have an INSERT handler without database code.
|
1 2 3 4 5 |
func (h *Handler) InsertDB(cx *gin.Context) { // Do database insert stuff h.notifier.Notify("acme", "INSERT", []byte(`{"id": 1, "name": acme}`)) cx.JSON(201, map[string]interface{}{"status": "ok"}) } |
“Do database insert stuff” could be for instance
|
1 2 3 4 5 6 7 |
ms := h.ms.Copy() defer ms.Close() acmeC := ms.DB("database").C("acme") model = new(Acme) cx.BindJSON(model) acmeC.Insert(model) |
and then you continue with notifying the connected clients via WebSocket by writing
|
1 2 3 4 |
modelJSON, _ := json.Marshal(model) h.notifier.Notify("acme", "INSERT", modelJSON) // and finally cx.JSON(201, nil) |
The very moment the notification is emitted it is sent(broadcast) to clients via WebSocket.
In your client-side webapp you receive that notification and filter by collection and do stuff like reloading entities or lists of entities.
By the way, if you don’t want to pass any data in the Notify function you can just write
|
1 |
h.notifier.Notify("acme", "INSERT", nil) |
Because you see the Notification type has the ,omitempty JSON tag.
The overhead of this whole ordeal is between 2-3 µs per notification.
You can also write
|
1 |
go h.notifier.Notify("acme", "INSERT", nil) |
if you’d like it to be done at the end, when your response finished.
Note the go in front of the Notify function
Now in case you were wondering how to use a notification by WebSocket, here’s an example for Angular 6.
First the base web-socket-service-ts
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
import {Injectable} from '@angular/core'; import {Observable, Observer, Subject} from 'rxjs'; @Injectable({ providedIn: 'root' }) export class WebSocketService { private subject: Subject<MessageEvent>; constructor() { } public connect(url: string): Subject<MessageEvent> { if (!this.subject) { this.subject = this.create(url); } return this.subject; } private create(url: string): Subject<MessageEvent> { const ws = new WebSocket(url); const observable = Observable.create( (obs: Observer<MessageEvent>) => { ws.onmessage = obs.next.bind(obs); ws.onerror = obs.error.bind(obs); ws.onclose = obs.complete.bind(obs); return ws.close.bind(ws); } ); const observer = { next: (data: Object) => { if (ws.readyState === WebSocket.OPEN) { ws.send(JSON.stringify(data)); } } }; return Subject.create(observer, observable); } } |
Building upon the WebSocketService is the NoficiationService
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
import {Injectable} from '@angular/core'; import {WebSocketService} from './web-socket.service'; import {environment} from '../../environments/environment'; import {map} from 'rxjs/operators'; import {Subject} from 'rxjs'; @Injectable({ providedIn: 'root' }) export class NotificationService { public messages: Subject<Notification>; constructor(ws: WebSocketService) { this.messages = <Subject<Notification>>ws.connect(environment.wsRoot).pipe( map((response: MessageEvent): Notification => { const data = JSON.parse(response.data); return { collection: data.collection, action: data.action, data: data.data }; }) ); } } export interface Notification { collection: string; action: string; data?: any; } |
And how you use this in your component, you do as you did before, you have an update data function where you grab the data from your endpoints, e.g.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
private update(): void { this.route.paramMap.subscribe(value => { const id = value.get('id'); this.id = id; this.communityService.find(id).subscribe(data => { this.community = data; this.containerService.list({community_id: data.id, preload_depth: '2'}).subscribe(containerdata => { this.containers = containerdata; }); }); }); } |
Let’s keep it simple for now, not go crazy with switchMap etc.
In NgOnInit then you initially call this update function but also subscribe to the notification service.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
ngOnInit() { this.update(); this.notifyService.messages.subscribe(data => { switch (data.collection) { case 'containers': case 'forums': { this.update(); } } }); } |
The above means, if the updated collection is either containers or forums, grab fresh data from the endpoints.
Don’t forget to unsubscribe in NgOnDestroy() in any case.
|
1 2 3 |
ngOnDestroy(): void { this.notifyService.messages.unsubscribe(); } |
So an example work-in-progress component.ts would look something like this
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
import {ActivatedRoute} from '@angular/router'; import {Community} from '../../models/community'; import {Container} from '../../models/container'; import {NotificationService} from '../../services/notification.service'; @Component({ selector: 'icod-community-home', templateUrl: './community-home.component.html', styleUrls: ['./community-home.component.scss'] }) export class CommunityHomeComponent implements OnInit, OnDestroy { private _id: string; private _community: Community = <Community>{}; private _containers: Container[] = []; constructor( private route: ActivatedRoute, private notifyService: NotificationService, private communityService: CommunityService, private containerService: ContainerService, ) { } ngOnInit() { this.update(); this.notifyService.messages.subscribe(data => { switch (data.collection) { case 'containers': case 'forums': { this.update(); } } }); } ngOnDestroy(): void { this.notifyService.messages.unsubscribe(); } private update(): void { this.route.paramMap.subscribe(value => { const id = value.get('id'); this.id = id; this.communityService.find(id).subscribe(data => { this.community = data; this.containerService.list({community_id: data.id, preload_depth: '2'}).subscribe(containerdata => { this.containers = containerdata; }); }); }); } get community() { return this._community; } set community(m: Community) { this._community = m; } get containers() { return this._containers; } set containers(m: Container[]) { this._containers = m; } get id(): string { return this._id; } set id(value: string) { this._id = value; } } |